Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for disabling serivce / system and disk offerings #9546

Open
wants to merge 16 commits into
base: main
Choose a base branch
from
Open

Add support for disabling serivce / system and disk offerings #9546

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
6932fb0
Add support for disabling serivce / system and disk offerings
Pearl1594 Aug 19, 2024
cfe4c6a
clean up virtual machines before cleaning up service offerings
Pearl1594 Aug 20, 2024
ea1c41d
Merge branch 'main' of https://github.com/apache/cloudstack into supp…
Pearl1594 Aug 20, 2024
6601b38
update api to list only active states by default, for backward compat…
Pearl1594 Aug 20, 2024
6ed1b60
fix tests and increase leniency
Pearl1594 Aug 22, 2024
fdb1f45
address comment
Pearl1594 Aug 22, 2024
3785bab
revert cleanup changes
Pearl1594 Aug 23, 2024
bd7ff55
fix 1 test
Pearl1594 Aug 23, 2024
9040055
address comments for test fixes
Pearl1594 Aug 26, 2024
d34a916
add new line
Pearl1594 Aug 26, 2024
1f3d2c9
fix test failures
Pearl1594 Aug 30, 2024
29eea23
detach volume for its cleanup
Pearl1594 Aug 30, 2024
bb8a660
fix few tests
Pearl1594 Aug 31, 2024
31f5c9b
Merge branch 'main' of https://github.com/apache/cloudstack into supp…
Pearl1594 Sep 10, 2024
a9f1b0a
fix tests
Pearl1594 Sep 10, 2024
590dc48
unused imports
Pearl1594 Sep 10, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 2 additions & 11 deletions api/src/main/java/org/apache/cloudstack/api/command/user/offering/ListDiskOfferingsCmd.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,7 @@
// under the License.
package org.apache.cloudstack.api.command.user.offering;

import static com.cloud.offering.DiskOffering.State.Active;

import com.cloud.offering.DiskOffering;
import org.apache.cloudstack.api.APICommand;
import org.apache.cloudstack.api.ApiConstants;
import org.apache.cloudstack.api.BaseListProjectAndAccountResourcesCmd;
Expand All @@ -29,7 +28,6 @@
import org.apache.cloudstack.api.response.VolumeResponse;
import org.apache.cloudstack.api.response.ZoneResponse;
import org.apache.commons.lang3.EnumUtils;
import org.apache.commons.lang3.StringUtils;

import com.cloud.offering.DiskOffering.State;

Expand Down Expand Up @@ -111,14 +109,7 @@
}

public State getState() {
if (StringUtils.isBlank(diskOfferingState)) {
return Active;
}
State state = EnumUtils.getEnumIgnoreCase(State.class, diskOfferingState);
if (!diskOfferingState.equalsIgnoreCase("all") && state == null) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these checks not relevant anymore?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just made this uniform to how it works with listing network and vpc offerings ; wherein when the state isn't passed, we list all offerings - so we do not explicitly need "all"

throw new IllegalArgumentException("Invalid state value: " + diskOfferingState);
}
return state;
return EnumUtils.getEnumIgnoreCase(DiskOffering.State.class, diskOfferingState);

Check warning on line 112 in api/src/main/java/org/apache/cloudstack/api/command/user/offering/ListDiskOfferingsCmd.java

View check run for this annotation

Codecov / codecov/patch

api/src/main/java/org/apache/cloudstack/api/command/user/offering/ListDiskOfferingsCmd.java#L112 

Added line #L112 was not covered by tests
}

public Long getVirtualMachineId() {
Expand Down
12 changes: 1 addition & 11 deletions ...rc/main/java/org/apache/cloudstack/api/command/user/offering/ListServiceOfferingsCmd.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,6 @@
// under the License.
package org.apache.cloudstack.api.command.user.offering;

import static com.cloud.offering.ServiceOffering.State.Active;

import org.apache.cloudstack.api.APICommand;
import org.apache.cloudstack.api.ApiConstants;
import org.apache.cloudstack.api.BaseListProjectAndAccountResourcesCmd;
Expand All @@ -28,7 +26,6 @@
import org.apache.cloudstack.api.response.UserVmResponse;
import org.apache.cloudstack.api.response.ZoneResponse;
import org.apache.commons.lang3.EnumUtils;
import org.apache.commons.lang3.StringUtils;

import com.cloud.offering.ServiceOffering.State;

Expand Down Expand Up @@ -157,14 +154,7 @@
}

public State getState() {
if (StringUtils.isBlank(serviceOfferingState)) {
return Active;
}
State state = EnumUtils.getEnumIgnoreCase(State.class, serviceOfferingState);
if (!serviceOfferingState.equalsIgnoreCase("all") && state == null) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just made this uniform to how it works with listing network and vpc offerings ; wherein when the state isn't passed, we list all offerings - so we do not explicitly need "all"

throw new IllegalArgumentException("Invalid state value: " + serviceOfferingState);
}
return state;
return EnumUtils.getEnumIgnoreCase(State.class, serviceOfferingState);

Check warning on line 157 in api/src/main/java/org/apache/cloudstack/api/command/user/offering/ListServiceOfferingsCmd.java

View check run for this annotation

Codecov / codecov/patch

api/src/main/java/org/apache/cloudstack/api/command/user/offering/ListServiceOfferingsCmd.java#L157 

Added line #L157 was not covered by tests
}

public Long getTemplateId() {
Expand Down
2 changes: 2 additions & 0 deletions engine/schema/src/main/java/com/cloud/vm/dao/VMInstanceDao.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -170,4 +170,6 @@ List<VMInstanceVO> searchRemovedByRemoveDate(final Date startDate, final Date en
List<Long> skippedVmIds);

Pair<List<VMInstanceVO>, Integer> listByVmsNotInClusterUsingPool(long clusterId, long poolId);

List<VMInstanceVO> listByOfferingId(long offeringId);
}
12 changes: 12 additions & 0 deletions engine/schema/src/main/java/com/cloud/vm/dao/VMInstanceDaoImpl.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,7 @@
protected SearchBuilder<VMInstanceVO> StartingWithNoHostSearch;
protected SearchBuilder<VMInstanceVO> NotMigratingSearch;
protected SearchBuilder<VMInstanceVO> BackupSearch;
protected SearchBuilder<VMInstanceVO> ServiceOfferingSearch;
protected SearchBuilder<VMInstanceVO> LastHostAndStatesSearch;
protected SearchBuilder<VMInstanceVO> VmsNotInClusterUsingPool;

Expand Down Expand Up @@ -325,6 +326,10 @@
VmsNotInClusterUsingPool.join("hostSearch2", hostSearch2, hostSearch2.entity().getId(), VmsNotInClusterUsingPool.entity().getHostId(), JoinType.INNER);
VmsNotInClusterUsingPool.and("vmStates", VmsNotInClusterUsingPool.entity().getState(), Op.IN);
VmsNotInClusterUsingPool.done();

ServiceOfferingSearch = createSearchBuilder();
ServiceOfferingSearch.and("serviceOfferingId", ServiceOfferingSearch.entity().getServiceOfferingId(), Op.EQ);
BackupSearch.done();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't it be ServiceOfferingSearch.done here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right

}

@Override
Expand Down Expand Up @@ -1069,4 +1074,11 @@
List<VMInstanceVO> uniqueVms = vms.stream().distinct().collect(Collectors.toList());
return new Pair<>(uniqueVms, uniqueVms.size());
}

@Override
public List<VMInstanceVO> listByOfferingId(long offeringId) {
SearchCriteria<VMInstanceVO> sc = ServiceOfferingSearch.create();
sc.setParameters("serviceOfferingId", offeringId);
return search(sc, null);
}

Check warning on line 1083 in engine/schema/src/main/java/com/cloud/vm/dao/VMInstanceDaoImpl.java

View check run for this annotation

Codecov / codecov/patch

engine/schema/src/main/java/com/cloud/vm/dao/VMInstanceDaoImpl.java#L1079-L1083 

Added lines #L1079 - L1083 were not covered by tests
}
11 changes: 7 additions & 4 deletions server/src/main/java/com/cloud/api/query/QueryManagerImpl.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3430,7 +3430,7 @@
diskOfferingSearch.and("computeOnly", diskOfferingSearch.entity().isComputeOnly(), Op.EQ);

if (state != null) {
diskOfferingSearch.and("state", diskOfferingSearch.entity().getState(), Op.EQ);
diskOfferingSearch.and("state", diskOfferingSearch.entity().getState(), Op.IN);

Check warning on line 3433 in server/src/main/java/com/cloud/api/query/QueryManagerImpl.java

View check run for this annotation

Codecov / codecov/patch

server/src/main/java/com/cloud/api/query/QueryManagerImpl.java#L3433 

Added line #L3433 was not covered by tests
}

// Keeping this logic consistent with domain specific zones
Expand Down Expand Up @@ -3547,6 +3547,8 @@

if (state != null) {
sc.setParameters("state", state);
} else {
sc.setParameters("state", Arrays.asList(ServiceOffering.State.Active, ServiceOffering.State.Inactive));

Check warning on line 3551 in server/src/main/java/com/cloud/api/query/QueryManagerImpl.java

View check run for this annotation

Codecov / codecov/patch

server/src/main/java/com/cloud/api/query/QueryManagerImpl.java#L3551 

Added line #L3551 was not covered by tests
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this required?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in case we don't specify what offering is required , this would list offerings in both active and inactive states

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you do not set the state condition (AKA where state in x), both active and inactive will be listed.

}

if (keyword != null) {
Expand Down Expand Up @@ -3756,7 +3758,7 @@
serviceOfferingSearch.select(null, Func.DISTINCT, serviceOfferingSearch.entity().getId()); // select distinct

if (state != null) {
serviceOfferingSearch.and("state", serviceOfferingSearch.entity().getState(), Op.EQ);
serviceOfferingSearch.and("state", serviceOfferingSearch.entity().getState(), Op.IN);

Check warning on line 3761 in server/src/main/java/com/cloud/api/query/QueryManagerImpl.java

View check run for this annotation

Codecov / codecov/patch

server/src/main/java/com/cloud/api/query/QueryManagerImpl.java#L3761 

Added line #L3761 was not covered by tests
}

if (vmId != null) {
Expand Down Expand Up @@ -3913,8 +3915,7 @@
}

serviceOfferingSearch.join("diskOfferingSearch", diskOfferingSearch, JoinBuilder.JoinType.INNER, JoinBuilder.JoinCondition.AND,
serviceOfferingSearch.entity().getDiskOfferingId(), diskOfferingSearch.entity().getId(),
serviceOfferingSearch.entity().setString("Active"), diskOfferingSearch.entity().getState());
serviceOfferingSearch.entity().getDiskOfferingId(), diskOfferingSearch.entity().getId());

Check warning on line 3918 in server/src/main/java/com/cloud/api/query/QueryManagerImpl.java

View check run for this annotation

Codecov / codecov/patch

server/src/main/java/com/cloud/api/query/QueryManagerImpl.java#L3918 

Added line #L3918 was not covered by tests
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't this a query that is executed if vmid != null? if so, should we list inactive offerings?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this PR aims to support disabling service offerings - to make it similar to the way network offerings work, it would make sense to list active and inactive offerings

Copy link
Contributor

@JoaoJandre JoaoJandre Aug 20, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method is too long (500+ lines...) for me to judge what it is exactly doing and where it is being done.
I just want to make sure that when creating VMs we don't list inactive offerings. If this change does not cause that, then I'm ok.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Al, that's a fair point.. Probably reverting to it listing Active by default would be a better option. Thanks.

}

if (cpuNumber != null) {
Expand Down Expand Up @@ -4043,6 +4044,8 @@
SearchCriteria<ServiceOfferingVO> sc = serviceOfferingSearch.create();
if (state != null) {
sc.setParameters("state", state);
} else {
sc.setParameters("state", Arrays.asList(ServiceOffering.State.Active, ServiceOffering.State.Inactive));

Check warning on line 4048 in server/src/main/java/com/cloud/api/query/QueryManagerImpl.java

View check run for this annotation

Codecov / codecov/patch

server/src/main/java/com/cloud/api/query/QueryManagerImpl.java#L4048 

Added line #L4048 was not covered by tests
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here, if you do not set this parameter, both active and inactive offerings will be queried.

}

if (vmId != null) {
Expand Down
18 changes: 12 additions & 6 deletions server/src/main/java/com/cloud/configuration/ConfigurationManagerImpl.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@
import com.cloud.host.HostTagVO;
import com.cloud.storage.StoragePoolTagVO;
import com.cloud.storage.VolumeApiServiceImpl;
import com.cloud.vm.VMInstanceVO;
import com.googlecode.ipv6.IPv6Address;
import org.apache.cloudstack.acl.SecurityChecker;
import org.apache.cloudstack.affinity.AffinityGroup;
Expand Down Expand Up @@ -4324,8 +4325,11 @@
}

annotationDao.removeByEntityType(AnnotationService.EntityType.DISK_OFFERING.name(), offering.getUuid());
offering.setState(DiskOffering.State.Inactive);
if (_diskOfferingDao.update(offering.getId(), offering)) {
List<VolumeVO> volumesUsingOffering = _volumeDao.findByDiskOfferingId(diskOfferingId);

Check warning on line 4328 in server/src/main/java/com/cloud/configuration/ConfigurationManagerImpl.java

View check run for this annotation

Codecov / codecov/patch

server/src/main/java/com/cloud/configuration/ConfigurationManagerImpl.java#L4328 

Added line #L4328 was not covered by tests
if (!volumesUsingOffering.isEmpty()) {
throw new InvalidParameterValueException(String.format("Unable to delete disk offering: %s [%s] because there are volumes using it", offering.getUuid(), offering.getName()));

Check warning on line 4330 in server/src/main/java/com/cloud/configuration/ConfigurationManagerImpl.java

View check run for this annotation

Codecov / codecov/patch

server/src/main/java/com/cloud/configuration/ConfigurationManagerImpl.java#L4330 

Added line #L4330 was not covered by tests
}
if (_diskOfferingDao.remove(offering.getId())) {
CallContext.current().setEventDetails("Disk offering id=" + diskOfferingId);
return true;
} else {
Expand Down Expand Up @@ -4397,15 +4401,17 @@
throw new InvalidParameterValueException(String.format("Unable to delete service offering: %s by user: %s because it is not root-admin or domain-admin", offering.getUuid(), user.getUuid()));
}

List<VMInstanceVO> vmsUsingOffering = _vmInstanceDao.listByOfferingId(offeringId);

Check warning on line 4404 in server/src/main/java/com/cloud/configuration/ConfigurationManagerImpl.java

View check run for this annotation

Codecov / codecov/patch

server/src/main/java/com/cloud/configuration/ConfigurationManagerImpl.java#L4404 

Added line #L4404 was not covered by tests
if (!vmsUsingOffering.isEmpty()) {
throw new CloudRuntimeException(String.format("Unable to delete service offering %s as it is in use", offering.getUuid()));

Check warning on line 4406 in server/src/main/java/com/cloud/configuration/ConfigurationManagerImpl.java

View check run for this annotation

Codecov / codecov/patch

server/src/main/java/com/cloud/configuration/ConfigurationManagerImpl.java#L4406 

Added line #L4406 was not covered by tests
}
annotationDao.removeByEntityType(AnnotationService.EntityType.SERVICE_OFFERING.name(), offering.getUuid());
if (diskOffering.isComputeOnly()) {
diskOffering.setState(DiskOffering.State.Inactive);
if (!_diskOfferingDao.update(diskOffering.getId(), diskOffering)) {
if (!_diskOfferingDao.remove(diskOffering.getId())) {
throw new CloudRuntimeException(String.format("Unable to delete disk offering %s mapped to the service offering %s", diskOffering.getUuid(), offering.getUuid()));
}
}
offering.setState(ServiceOffering.State.Inactive);
if (_serviceOfferingDao.update(offeringId, offering)) {
if (_serviceOfferingDao.remove(offeringId)) {
CallContext.current().setEventDetails("Service offering id=" + offeringId);
return true;
} else {
Expand Down
13 changes: 12 additions & 1 deletion test/integration/component/test_acl_listvolume.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -340,8 +340,19 @@ def setUpClass(cls):
cls.vm_a_volume = Volume.list(cls.apiclient, virtualmachineid=cls.vm_a.id)

cls.cleanup = [
cls.vm_d1,
cls.vm_d1a,
cls.vm_d1b,
cls.vm_d11,
cls.vm_d11a,
cls.vm_d11b,
cls.vm_d111a,
cls.vm_d12a,
cls.vm_d12b,
cls.vm_d2,
cls.vm_a
cls.account_a,
cls.service_offering,
cls.service_offering
DaanHoogland marked this conversation as resolved.
Show resolved Hide resolved
]
except Exception as e:
cls.domain_2.delete(cls.apiclient, cleanup="true")
Expand Down
13 changes: 7 additions & 6 deletions test/integration/component/test_cpu_max_limits.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -230,7 +230,6 @@ def test_02_deploy_vm_account_limit_reached(self):
self.testdata["service_offering_multiple_cores"]
)
# Adding to cleanup list after execution
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# Adding to cleanup list after execution

self.cleanup.append(self.service_offering)

self.debug("Setting up account and domain hierarchy")
self.setupAccounts(account_limit=6, domain_limit=8)
Expand All @@ -242,8 +241,10 @@ def test_02_deploy_vm_account_limit_reached(self):
self.debug("Deploying instance with account: %s" %
self.child_do_admin.name)

self.createInstance(account=self.child_do_admin,
self.vm1 = self.createInstance(account=self.child_do_admin,
service_off=self.service_offering, api_client=api_client_admin)
self.cleanup.append(self.vm1)
self.cleanup.append(self.service_offering)
DaanHoogland marked this conversation as resolved.
Show resolved Hide resolved
DaanHoogland marked this conversation as resolved.
Show resolved Hide resolved

self.debug("Deploying instance when CPU limit is reached in account")

Expand Down Expand Up @@ -305,8 +306,6 @@ def test_04_deployVm__account_limit_reached(self):
self.apiclient,
self.testdata["service_offering_multiple_cores"]
)
# Adding to cleanup list after execution
self.cleanup.append(self.service_offering)

self.debug("Setting up account and domain hierarchy")
self.setupAccounts(account_limit=5, domain_limit=5, project_limit=5)
Expand All @@ -317,9 +316,11 @@ def test_04_deployVm__account_limit_reached(self):

self.debug("Deploying instance with account: %s" %
self.child_do_admin.name)
self.createInstance(account=self.child_do_admin,
self.vm2 = self.createInstance(account=self.child_do_admin,
service_off=self.service_offering, api_client=api_client_admin)

# Adding to cleanup list after execution
self.cleanup.append(self.vm2)
self.cleanup.append(self.service_offering)
DaanHoogland marked this conversation as resolved.
Show resolved Hide resolved
self.debug("Deploying instance in project when CPU limit is reached in account")

with self.assertRaises(Exception):
Expand Down
4 changes: 3 additions & 1 deletion test/integration/component/test_network_offering.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1449,14 +1449,16 @@ def test_create_network_with_snat(self):

self.debug("Deploying VM in account: %s on the network %s" % (self.account.name, self.network.id))
# Spawn an instance in that network
VirtualMachine.create(
self.vm1 = VirtualMachine.create(
self.apiclient,
self.services["virtual_machine"],
accountid=self.account.name,
domainid=self.account.domainid,
serviceofferingid=self.service_offering.id,
networkids=[str(self.network.id)]
)
self.cleanup.append(self.vm1)
self.cleanup.append(self.network)
DaanHoogland marked this conversation as resolved.
Show resolved Hide resolved
self.debug("Deployed VM in network: %s" % self.network.id)

src_nat_list = PublicIPAddress.list(
Expand Down
2 changes: 1 addition & 1 deletion test/integration/component/test_stopped_vm.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1444,7 +1444,7 @@ def test_vm_per_account(self):
serviceofferingid=self.service_offering.id,
startvm=False
)

self.cleanup.append(virtual_machine)
# Verify VM state
self.assertEqual(
virtual_machine.state,
Expand Down
Loading
Loading