07070100081015000081A40000000000000000000000015B64B4DF0000004E000000FD0000000200000000000000000000004500000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/.copyrightignore.copyrightignore .rsync-filter requirements.txt test-requirements.txt tox.ini 07070100081058000081A40000000000000000000000015B64B4DF00000080000000FD0000000200000000000000000000003F00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/.gitreview[gerrit] host=gerrit.suse.provo.cloud port=29418 project=ardana/opsconsole-server.git defaultremote=ardana defaultbranch=master 07070100081036000081A40000000000000000000000015B64B4DF00000B5D000000FD0000000200000000000000000000004000000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/HACKING.rst.. (c) Copyright 2016 Hewlett Packard Enterprise Development LP (c) Copyright 2017 SUSE LLC Creating a New Service ---------------------- In order to create a new service, perform the following steps: #. Add an entry_point for your service to the ``entry_points`` section of ``setup.py``. This is important since this is how Stevedore will locate, load, and start the service. The ``entry`` function can be used to register the filename and the classname. For example:: entry_point={'bll.plugins': [ # existing entries ... entry("nova", "NovaSvc"), ], }, #. Inherit from ``bll.plugins.service.SvcBase`` and implement methods. Annotate your methods with the ``@expose`` decorator, and the function will automatically be callable based on the name of the operation and/or action. #. Document you methods using standard python docstrings per https://www.python.org/dev/peps/pep-0257/ . It is especially important to document the parameters that each exposed method expects. #. Add unit tests to ``tests/plugins``. Every new feature should have unit tests created that test the feature. Running Tests ------------- "Functional" tests are tests that use the python unittest library and which require some external resource, preventing them from being run on a normal CI build that does not have these resources available. A typical example of this is a plugin that requires keystone to work properly. The ideal way to test these functions is to use mocks, but that often is impractical or will result in meaningless tests. Since unit tests also use the same unittest python library, what distinguishes a functional test is that it has a ``@functional`` decorator on the class or method. Any class with this decoration will be skipped unless the environment variable named ``functional`` contains the string in question. For example, to exercise all functional tests that can be run in the current environment based on the services present, you can either do:: env functional=auto ./run_tests.sh or equivalently:: ./run_tests.sh -f auto If, however, you want to explicitly control which tests should be run, you can also specify the dependent services, for example:: env functional=keystone ./run_tests.sh When explicitly specifying dependent services, you are responsible for making those services available before running the tests. Specific tests can be run by giving an additional argument listing the unit tests module/class/function to run, using the syntax specifid in https://docs.python.org/2/library/unittest.html#command-line-interface Coverage reports can be generated by using the ``-c`` argument to run_tests. Building Docs ------------- Normal Sphinx docs can be built with ``run_tests.sh --docs``. The resulting html documents will be created and put into the doc/build subdirectory. 07070100080FF0000081A40000000000000000000000015B64B4DF0000279F000000FD0000000200000000000000000000003C00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/LICENSE Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. 0707010008105E000081A40000000000000000000000015B64B4DF000000C9000000FD0000000200000000000000000000004000000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/MANIFEST.in# (c) Copyright 2015-2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC # Add items not already included by the 'packages' clause in setup.py recursive-include bll *.sql *.mo 07070100081059000081A40000000000000000000000015B64B4DF00001B1D000000FD0000000200000000000000000000003F00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/README.rst.. (c) Copyright 2015-2016 Hewlett Packard Enterprise Development LP (c) Copyright 2017-2018 SUSE LLC ======== Overview ======== Basic Architecture ------------------ The Business Logic Layer (BLL) is the server component of the operations console. It consists of a single component which handles all incoming requests and executes each in a thread. It runs under the pecan_ WSGI framework. In production, the web layer is hosted under Apache, but it can also run standalone for development and testing. .. _pecan: http://pypi.python.org/pypi/pecan Configuration Files ------------------- The configuration for the BLL is contained in a Python-formatted text file. There have been many copies of this file floating around inside and outside of the source tree, used for the variety of ways that the BLL can be launched. Now there is just a single copy file in this source distribution, ``tests/config.py``, that is used for the BLL testing. The production version of this file is maintained in a separate repo. In modern versions of pecan (0.8.3. and above), the config file is required to be a python file whose name ends with ``.py``, and will fail at startup if it is anything else, including ``.conf``. Launch environments ................... There are a number of ways the code can be run: - Unit/functional tests Launched via run_tests.sh or via an IDE. The Jenkins build also launches functional tests via run_tests.sh. Note: since Python tests do not have a "main" program, there is no way to pass command line parameters, and thus the only way to change which config file used is via the BLL_CONF_OVERRIDE environment variable. - ``run_tests.sh --runserver`` Launches the pecan web server directly, using the config file name as a command line argument. - apache / wsgi In production, the web layer runs in a pecan container under apache, via WSGI. The file that is launched is ``app.wsgi``, and it in turn deploys pecan and specifies the config file to be used. Note: ``app.wsgi`` is generated by ansible from a template. Config file properties ...................... The config file, ``tests/config.py``, used for the BLL development and testing is intended to be flexible enough for the variety of situations in which it is used and normally requires no local changes. These properties include: - console logging All logging is done to the console, which is the most useful place for dev/test. By not logging to the file system, it also avoids the issue of finding a system-independent, launch-independent, writable location to place the file(s). Of course, standard shell redirection and/or tee-ing can be used to save the output into a file. - environment variables Most values that are dependent upon the environment and are likely to change are first looked up in the environment, and use a reasonable default if not set. In the event that some value in this file needs to be customized for a particular environment, the first (and preferred) option is to add another configuration variable within the file and use a reasonable default for it. If that is not possible, then the next best choice is to copy the file somewhere else (usually outside of the source tree), make the local modifications, and then specify that the bll use this file for configuration. There are a couple ways to specify this, as demonstrated in the next section. Resolution order ................ The BLL figures out which config file to use based on a resolution order that is, as much as possible and reasonable, common to all ways of launching it, with exceptions noted below. The BLL will use the *first* file it can find, and will stop searching for others; therefore do *not* expect to be able to set some values in one config file and other values in another and expect it to read them both and merge them together at run-time. The order as follows: #. Filename passed in as a command-line arguments This is supported in those launchers that accept command line arguments: the test script and ``run_tests.sh`` #. Filename specified in BLL_CONF_OVERRIDE environment variable Specifies the config file to use. Supported everywhere. #. File from source distribution When running in a dev/test environment, i.e. via ``run_tests.sh`` or via the unit/functional test framework, the file ``tests/config.py`` will be used. Note: this file is not present on a production system, and the launchers used in production (via apache) will not look for this file. #. ``/etc/opsconsole-server/config.py`` (production only) When launched in production (via apache) this file is used if present. #. ``/etc/opsconsole-server/opsconsole-server.conf`` (production only -- deprecated) When launched in production (via apache) this file is used as a last resort. Since its name prevents migrating to a modern version of pecan, it is expected that this will be removed in an upcoming release. Localization ------------ Message files ............. Message files are created and managed in several steps. The first is to extract strings from all python code that calls the ``_()`` or ``gettext()`` functions. These strings are placed into the Portable Object Template (``.pot``) file. To create or update this ``.pot`` file:: ./setup.py extract_messages -o bll/locale/messages.pot Tweak the copyright that is inserted into the generated file, ``bll/locale/messages.pot`` to match the corporate standard. This extraction step needs to be done whenever localizable strings in python code are added or modified. Portable Object (``.po``) files are created for each locale from the ``.pot`` file. These ``.po`` files contain translations for each of the strings. These files are sent off to the translators for translation, and the translated files should be checked back into the git repository. Whenever the ``.pot`` template file is changed, the ``.po`` files can be updated with:: ./setup.py update_catalog -d bll/locale -i bll/locale/messages.pot The final step is to create Machine Object (``.mo``) files from the ``.po`` files, which are used at runtime. These files are automatically generated by the continuous integrated build process via ``setup.py bdist``. If you want to create these ``.mo`` files manually for development and testing, use:: ./setup.py compile_catalog -d bll/locale -f For more information about these working with message catalogs and the ``setup`` commands, see the babel_ page. .. _babel: http://babel.pocoo.org/en/latest/messages.html Python usage ............ To use strings in plugin code:: raise BllException(self._("Error message")) To use strings with a single placeholder:: raise BllException(self._("Error with id {}").format(id)) To use with multiple placeholders:: raise BllException(self._("Error with id {1} doing operation {2}").format( id, operation)) 07070100081061000081A40000000000000000000000015B64B4DF00000010000000FD0000000200000000000000000000003E00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/babel.cfg[python: **.py] 070701000DFF80000041ED0000000000000000000000065B64B4DF00000000000000FD0000000200000000000000000000003800000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/bll070701000DFFAD000081A40000000000000000000000015B64B4DF00000000000000FD0000000200000000000000000000004400000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/bll/__init__.py070701000DFF91000041ED0000000000000000000000035B64B4DF00000000000000FD0000000200000000000000000000003C00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/bll/api070701000DFF92000081A40000000000000000000000015B64B4DF00000334000000FD0000000200000000000000000000004800000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/bll/api/__init__.py# (c) Copyright 2015-2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC ACTION = 'action' AUTH_TOKEN = 'auth_token' COMPLETE = 'complete' DATA = 'data' DURATION = 'duration' ENDTIME = 'endtime' JOB_STATUS_REQUEST = 'job_status_request' LANGUAGE = 'language' OPERATION = 'operation' PATH = 'path' PERCENT_COMPLETE = 'percentComplete' POLLING_INTERVAL = 'recommendedPollingInterval' PROGRESS = 'progress' REGION = 'region' REQUEST_DATA = 'request_data' REQUEST_ID = 'request_id' REQUEST_PARAMETERS = 'request_parameters' STARTTIME = 'starttime' STATUS = 'status' STATUS_ERROR = 'error' STATUS_INPROGRESS = 'inprogress' STATUS_NOT_FOUND = 'not_found' STATUS_WARNING = 'warning' TARGET = 'target' TENANT_ID = 'tenant_id' TXN_ID = 'txn_id' VERSION = 'api_version' USER_AGENT = "Operations Console" 070701000DFF94000081A40000000000000000000000015B64B4DF00003040000000FD0000000200000000000000000000004A00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/bll/api/auth_token.py# # (c) Copyright 2015-2017 Hewlett Packard Enterprise Development LP # (c) Copyright 2017-2018 SUSE LLC # import logging import warnings from dogpile.cache import make_region from dogpile.cache.util import sha1_mangle_key from keystoneclient.auth.identity import v3 from keystoneclient import session, exceptions from keystoneclient.v3 import client as ksclient3 from requests.packages.urllib3.exceptions import InsecureRequestWarning from bll.api import USER_AGENT from bll.common.exception import BllAuthenticationFailedException from bll.common.util import get_conf, start_cache_cleaner LOG = logging.getLogger(__name__) # There are two separate caches being created. # # The "static" cache contains items that are unlikely to change while the BLL # is run and whose number of entries is not going to continue to grow. These # items are not specific to the logged in user. # # The "session" cache contains items that are tied to keystone sessions, # and the region's timeout is set to match keystone's (4 hours). After # the expiration time is hit, dogpile will not return the value from the cache, # but will trigger the function to run and re-obtain its values. The # in-memory backends supplied by dogpile do not actually delete expired # entries from the cache, so a separate thread is spawned to periodically # clean these up to avoid runaway memory usage. # static = make_region(key_mangler=sha1_mangle_key).configure( 'dogpile.cache.memory') cache = {} cache_expiration = get_conf('cache_expiration', 14400) # 4 hours # session = make_region(key_mangler=sha1_mangle_key).configure( session_cache = make_region().configure( 'dogpile.cache.memory', expiration_time=cache_expiration, arguments={"cache_dict": cache}) start_cache_cleaner(cache, cache_expiration, "SessionCacheCleaner") def login(username, password, domain='Default'): """ Perform the initial login to the BLL, using the given credentials. This uses the "normal" keystone workflow of: - Connecting to keystone and obtain an unscoped token (not scoped to any particular project or domain) - Get a list of projects available to the given user - Select a project, and connect to keystone again (with the unscoped token) to receive a project-scoped token. Returns a keystone auth_ref, which contains the token, expiration, and other info about the authentication This function is primarily used in the initial login UI screen, but is also used by the deploy service """ LOG.debug("Obtaining unscoped token for user %s", username) auth = v3.Password(auth_url=get_auth_url(), username=username, password=password, user_domain_name=domain, unscoped=True) unscoped_session = session.Session(auth=auth, user_agent=USER_AGENT, verify=verify()) try: unscoped_token = unscoped_session.get_token() except Exception as e: raise BllAuthenticationFailedException( "User is not authorized on the %s domain: [%s]" % (domain, str(e))) return get_appropriate_auth_ref(unscoped_token) def validate(token): return get_appropriate_auth_ref(token) is not None @session_cache.cache_on_arguments() def get_appropriate_auth_ref(token): """ The incoming token does not indicate which project it is for. Therefore we find the appropriate project (normally one for which the token's user has admin privilege for) and return an auth_ref. The auth_ref can then be used to instantiate a Keystone object without requiring an additional round-trip to authenticate against keystone """ LOG.debug("Obtaining unscoped keystone client with token") auth = v3.Token(auth_url=get_auth_url(), token=token, unscoped=True) sess = session.Session(auth=auth, user_agent=USER_AGENT, verify=verify()) ks = ksclient3.Client(session=sess, user_agent=USER_AGENT) project_list = [t.name for t in ks.projects.list(user=sess.get_user_id())] auth_ref = _find_appropriate_project(token, project_list) # Verify that the user is a 'cloud admin', i.e. that they have the # admin role on the default domain. Domain roles are only valid in # keystone v3, so make sure to use an appropriate auth URL role_names = [] try: ks = ksclient3.Client(token=token, auth_url=get_auth_url(), domain_name='default', verify=verify()) role_names = ks.auth_ref.role_names except Exception: raise BllAuthenticationFailedException( "User is not authorized on the default domain") if 'admin' not in role_names: raise BllAuthenticationFailedException( "User is not an admin of the default domain") return auth_ref def _find_appropriate_project(token, project_list): # sort and order the list so that admin is the first project to try, # since that is the one that most administrators have the admin role in if 'admin' not in project_list: raise BllAuthenticationFailedException( "User does not have access to the admin project") # Users must have the admin or monasca-user role on the admin project # in order to manage monasca entities auth_ref = _get_auth_ref(token, 'admin') if 'admin' in auth_ref.role_names: return auth_ref if 'monasca-user' not in auth_ref.role_names: raise BllAuthenticationFailedException( "User does not have the proper role in the admin project") # Since we have already authorized against the admin project, there is # no need to do it again in the following loop projects = sorted(project_list) projects.remove('admin') # We do not know which project the token has admin role on, # so we have to check against the list of projects for project in projects: auth_ref = _get_auth_ref(token, project) if 'admin' in auth_ref.role_names: return auth_ref # If we got here, we do not have admin access raise BllAuthenticationFailedException( "User does not have admin access") @session_cache.cache_on_arguments() def _get_auth_ref(token, project_name, domain_name='Default'): """ Return auth ref for the given token and project_name """ LOG.debug("Obtaining scoped keystone client with token, project %s", project_name) auth = v3.Token(auth_url=get_auth_url(), project_name=project_name, project_domain_name=domain_name, token=token) project_session = session.Session(auth=auth, verify=verify(), user_agent=USER_AGENT) # Trigger the generation of the new token project_session.get_auth_headers() return project_session.auth.auth_ref def get_auth_url(version='v3'): """ Get default url from config file :return: """ url = get_conf("keystone.private_url") if url is None: url = '%s://%s:%d' % ( get_conf("keystone.protocol"), get_conf("keystone.host"), get_conf("keystone.public_port")) return '%s/%s' % (url, version) @session_cache.cache_on_arguments() def _get_session(token): """ Return v3 session for token """ auth_ref = get_appropriate_auth_ref(token) auth = v3.Token(auth_url=get_auth_url(), project_id=auth_ref.project_id, token=token) return session.Session(auth=auth, user_agent=USER_AGENT, verify=verify()) def _get_domain_session(token, domain_name=None): """ Return v3 session for token """ domain_name = domain_name or 'default' auth = v3.Token(auth_url=get_auth_url(), domain_id=domain_name, token=token) return session.Session(auth=auth, user_agent=USER_AGENT, verify=verify()) warnings_filtered = False def verify(): global warnings_filtered # Perform SSL verification unless explicitly configured otherwise insecure = get_conf("insecure") verify = False if insecure else get_conf("keystone.cacert") # If SSL verification is turned off, just log the insecure warning once # instead of repeating it ad nauseum. Use warnings_filtered # boolean to avoid repeatedly adding a filter to the warnings system if not verify and not warnings_filtered: warnings.filterwarnings("once", category=InsecureRequestWarning) warnings_filtered = True return verify class TokenHelpers: def __init__(self, token): self.token = token def get_user_token(self): """ Return the original token that the user authenticated with """ return self.token def get_token_for_project(self, project_name): """ Get the token for a specific project. This is needed in places where self.token corresponds to a project other than the one that is needed by the caller. """ return _get_auth_ref(self.token, project_name).auth_token @static.cache_on_arguments() def get_endpoints(self, service_type, region=None, endpoint_type=get_conf("services.endpoint_type", default="internalURL")): """ Returns a list of unique internal service endpoint(s) for the given service type and region. If no region is specified, keystone will capture the unique endpoints across regions. If two regions share the same URL, then only one will be returned. For example, if keystone contains 3 endpoints for service x: [ { 'region:'1', 'service_type':'x', 'url':'http://y' }, { 'region:'2', 'service_type':'x', 'url':'http://y' }] then only one entry will be returned since they share a common URL: [ { 'region:'1', 'service_type':'x', 'url':'http://y' }] """ auth_ref = get_appropriate_auth_ref(self.token) urls = auth_ref.service_catalog.get_endpoints( service_type=service_type, endpoint_type=endpoint_type, region_name=region) if not urls or not urls[service_type]: return [] # Return only one endpoint per unique URL return {u['url']: u for u in urls[service_type]}.values() @static.cache_on_arguments() def get_service_endpoint(self, service_type, endpoint_type=get_conf("services.endpoint_type", default="internalURL"), region=None): """ Get the internal service endpoint for the given service type. In a multi-region environment where no region is passed, only a single endpoint will be returned. """ auth_ref = get_appropriate_auth_ref(self.token) try: return auth_ref.service_catalog.url_for( service_type=service_type, endpoint_type=endpoint_type, region_name=region) except exceptions.EndpointNotFound: pass def get_session(self): """ Get a project-scoped session. """ return _get_session(self.token) def get_domain_session(self): """ Get a domain-scoped session. It is expected that this will only be used by plugins that need to manipulate keystone users. """ return _get_domain_session(self.token) @static.cache_on_arguments() def get_regions(self): """ Obtain a list of regions available in the current environment. """ client = ksclient3.Client(session=self.get_session(), endpoint_type=get_conf( "services.endpoint_type", default="internalURL"), user_agent=USER_AGENT) return client.regions.list() 070701000DFF96000041ED0000000000000000000000025B64B4DF00000000000000FD0000000200000000000000000000004800000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/bll/api/controllers070701000DFF9A000081A40000000000000000000000015B64B4DF00000000000000FD0000000200000000000000000000005400000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/bll/api/controllers/__init__.py070701000DFF9E000081A40000000000000000000000015B64B4DF00001319000000FD0000000200000000000000000000005A00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/bll/api/controllers/app_controller.py# # (c) Copyright 2015-2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC # import logging import json from pecan import response, request, expose from pecan.rest import RestController from bll.api.request import BllRequest from bll import api from bll.common.util import context, scrub_passwords, response_to_string from bll.common.job_status import get_job_status from bll.plugins.service import SvcBase LOG = logging.getLogger(__name__) region = None class AppController(RestController): @expose('json') def post(self, **kwargs): # generate a uniq id solely for the purpose of more # easily matching up request/responses in the log try: bll_request = BllRequest(json.loads(request.body)) # Add to thread local storage for logging context.txn_id = bll_request.txn_id if 'X-Auth-Token' in request.headers: bll_request[api.AUTH_TOKEN] = request.headers['X-Auth-Token'] bll_request[api.LANGUAGE] = self.get_language( request.headers.get('Accept-Language')) LOG.info("Received %s", bll_request) # initial service request? if bll_request.is_service_request(): ret = SvcBase.spawn_service(bll_request) else: # Poll to retrieve async response ret = get_job_status(bll_request.txn_id) response.status = 201 if isinstance(ret, dict): logstr = response_to_string(ret) LOG.info("Response %s", logstr) except ValueError as info: # json.loads was unable to convert the request to json LOG.error("Error converting request body to json: %s. " "Request body: %s", info, request.body) response.status = 400 ret = { api.STATUS: api.STATUS_ERROR, api.DATA: [{api.DATA: str(info)}] } LOG.info("Response ValueError: %s", scrub_passwords(ret)) except Exception as info: response.status = 400 ret = { api.STATUS: api.STATUS_ERROR, api.DATA: [{api.DATA: str(info)}] } LOG.info("Response Exception: %s", scrub_passwords(ret)) # Clear out txn_id as it leaves the system context.txn_id = '' return ret def get_language(self, accept_language): # Obtains the set of languages from the parameter (Accept-Language # http header) and returns the best match against those languages # that are available. # # The language parameter is a comma-separated list of languages. Each # language may contain a quality value, which is a string like # ;q=0.5 that is appended to the language name. For example: # da, en-gb;q=0.8, en;q=0.7 AVAILABLE_LANGUAGES = ['en', 'ja', 'zh'] DEFAULT_LANGAUAGE = 'en' if accept_language: accept_language = accept_language.replace(' ', '') # Extract languages and their priorities from the Accept-Language # header. Using the above example string, this logic would # populate the languages array with: # [ ('da', 1.0), ('en-gb', 0.8), ('en', 0.7) ] languages = [] for language_str in accept_language.split(','): parts = language_str.split(';q=') language = parts[0] if len(parts) > 1: priority = float(parts[1]) else: priority = 1.0 languages.append((language, priority)) # Sort languages according to preference (highest first) languages.sort(key=lambda t: t[1], reverse=True) # Find first exact match against the available languages. In # the above example, if en-gb was in Accept-Languages, a match # against an available language of 'en-GB' should be preferred # above a match against the generic 'en'. In other words, if # the user wants British english and it is available, it should # be used instead of the generic (likely US) English translation. for want, priority in languages: for avail in AVAILABLE_LANGUAGES: if want.replace('-', '_').lower() == avail.lower(): return avail # Fall back to find first language match (prefix e.g. 'en' for # 'en-GB') for want, priority in languages: for avail in AVAILABLE_LANGUAGES: if want.split('-')[0].lower() == avail.lower(): return avail return DEFAULT_LANGAUAGE 070701000DFF98000081A40000000000000000000000015B64B4DF000001C4000000FD0000000200000000000000000000005000000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/bll/api/controllers/root.py# # (c) Copyright 2015-2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC # from pecan import expose from v1 import V1 class RootController(object): v1 = V1() # Return a result when querying the root document. @expose(generic=True, template=None, content_type='text/html') @expose(generic=True, template=None, content_type='application/json') def index(self): return '"Operations Console API"' 070701000DFF9D000081A40000000000000000000000015B64B4DF00000BEB000000FD0000000200000000000000000000004E00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/bll/api/controllers/v1.py# (c) Copyright 2015-2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC import json import logging from pecan import expose, request from pecan.core import Response from pecan.secure import unlocked, SecureController from bll.api.auth_token import login, validate from bll.api.controllers.app_controller import AppController LOG = logging.getLogger(__name__) class V1(SecureController): _custom_actions = { 'auth_token': ['POST'] } def __init__(self, *args, **kwargs): super(V1, self).__init__(*args, **kwargs) @classmethod def check_permissions(self): """ Since this object extends SecureController, this function will automatically be called on all exposed rest calls, except for those decorated as @unlocked or decorated with a custom @secure function """ req_body = None try: req_body = json.loads(request.body) # Bypass deploy permission checks if req_body['target'] == 'eula': LOG.debug("v1 bypass check_permissions for %s", req_body['target']) return True except Exception: pass if 'X-Auth-Token' not in request.headers: return False token = request.headers['X-Auth-Token'] # For backward compatibility with old HDP installers that use the # old token blob, extract the token from the blob if isinstance(req_body, dict) and req_body.get('target') == 'plugins'\ and "management_appliance" in token: try: blob = json.loads(token) ma_tokens = blob['management_appliance']['tokens'] token = ma_tokens[0]['auth_token'] request.headers['X-Auth-Token'] = token except Exception: pass return validate(token) @unlocked @expose(content_type='application/json') def auth_token(self, username=None, password=None, tenant=None): """ POST /auth_token BODY {"username": username, "password": password} """ body = json.loads(request.body) username = body['username'] password = body['password'] LOG.debug("/auth_token user=%s" % (username)) auth_ref = None try: auth_ref = login(username, password) return json.dumps({ 'token': auth_ref.auth_token, 'expires': auth_ref.expires.isoformat() }) except Exception as e: LOG.info("User login as %s failed: %s" % (username, str(e))) return Response(str(e), 401) @unlocked # Return a result when querying the root document. @expose(generic=True, template=None, content_type='text/html') @expose(generic=True, template=None, content_type='application/json') def index(self): return '"Operations Console API v1"' # V1 App Controller bll = AppController() 070701000DFF93000081A40000000000000000000000015B64B4DF00001822000000FD0000000200000000000000000000004700000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/bll/api/request.py# (c) Copyright 2015-2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC import operator from bll.common.util import scrub_passwords, new_txn_id from bll.common.exception import InvalidBllRequestException from bll import api class BllRequest(dict): RESERVED = (api.TARGET, api.ACTION, api.TXN_ID, api.REGION, api.AUTH_TOKEN, api.DATA, api.LANGUAGE) def __init__(self, request=None, target=None, auth_token=None, operation=None, action=None, data=None, txn_id=None, region=None, language=None, **kwargs): """ Create the BLL request. Requests are generally created in one of two ways: 1. With only a request parameter - These include calls from the external REST API (where request is populated with the JSON payload) and from older tests and cross-plugin calls 2. Explicit parameters and no request parameter - These are mostly from tests and plugins BllRequest is just a slightly enhanced dictionary. Historically the information specific to a particular request has been passed as a nested dictionary whose key is 'data'. While this avoids potential conflicts between RESERVED key names and those needed by the given operation, in practice this has never been an issue. The down side is that the request JSON has unnecessary nesting, which makes reading requests more confusing and requires extra code to navigate down into the nested dictionary. To make matters worse, a convention has arisen in a few services to expect their data in another dictionary nested within the 'data' dictionary, whose key name is 'data' or 'request_data'. Therefore these APIs have doubly nested dictionary. This is just crazy and must be fixed. Going forward, callers should place all parameters at the top level of the request rather than embedding them in a nested dictionary. In order to provide backward compatibility for older UI calls (which still populates the 'data' dictionary) and older BLL plugins (which still expect 'data' to be populated), this constructor will yield a request object where all non-RESERVED request fields will be populated as both top-level elements and as elements in the data dictionary. """ if request: # When called via the external API, request will be dictionary # populated with a dictionary populated from the JSON payload. super(BllRequest, self).__init__(request) if kwargs: self.update(**kwargs) if not self.get(api.DATA): self[api.DATA] = {} if target: self[api.TARGET] = target if isinstance(data, dict): self[api.DATA].update(data) # Copy the operation into the data dict. The operation # may either be in a specific argument (when called from other plugins) # or already populated in (when called from external REST # API) op = operation or self.get(api.OPERATION) if op: self[api.DATA][api.OPERATION] = op # Provide backward compatibility to older callers by copying # everything from the data dictionary into top-level elements of the # request. for key, value in self[api.DATA].iteritems(): if key not in self.RESERVED: self[key] = value # Provide backward compatibility to older services by copying # everything from top level of the request into the data # dictionary for key, value in self.iteritems(): if key not in self.RESERVED and key not in self[api.DATA]: self[api.DATA][key] = value if action: self[api.ACTION] = action if auth_token: self[api.AUTH_TOKEN] = auth_token if txn_id: self[api.TXN_ID] = txn_id if region: self[api.REGION] = region if language: self[api.LANGUAGE] = language self._verify_request() if not self.get(api.TXN_ID): self[api.TXN_ID] = new_txn_id() self.txn_id = self[api.TXN_ID] def _verify_request(self): if len(self) == 0: raise InvalidBllRequestException('No request') # txn_id is required when requesting a job status update if self.is_job_status_request() and not self.get(api.TXN_ID): raise InvalidBllRequestException('No txn_id') def is_service_request(self): return not self.is_job_status_request() def is_job_status_request(self): return self.get(api.JOB_STATUS_REQUEST, False) def get_data(self): """ return a dictionary of all items from data except 'operation', 'version', and any RESERVED words """ excluded = self.RESERVED + (api.OPERATION, api.VERSION) return {k: v for k, v in self.iteritems() if k not in excluded} def __str__(self): # Print the DATA portion of the request with the OPERATION first, # and other keys afterward in sorted order data = self.get(api.DATA) data_list = [] text = "" if api.TARGET in self: text += "TARGET:%s " % self[api.TARGET] if self.is_job_status_request(): text += "STATUS_REQUEST " if api.ACTION in self: text += "ACTION:%s " % self[api.ACTION] if isinstance(data, dict): sorted_keys = sorted(data, key=operator.itemgetter(1)) if api.OPERATION in sorted_keys: sorted_keys.remove(api.OPERATION) text += "OPERATION:%s " % data[api.OPERATION] for key in sorted_keys: value = data[key] data_list.append("%s:%s" % (key, scrub_passwords(value))) text += scrub_passwords("DATA:{%s} " % ",".join(data_list)) if api.TXN_ID in self: text += "TXN:%s " % self[api.TXN_ID] return text 070701000DFF95000081A40000000000000000000000015B64B4DF0000061C000000FD0000000200000000000000000000004800000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/bll/api/response.py# (c) Copyright 2015-2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC # # Created on Dec 16, 2014 import time from bll import api from bll.common.util import response_to_string class BllResponse(dict): ''' classdocs ''' def __init__(self, bll_request): super(BllResponse, self).__init__() ''' Constructor ''' self[api.STARTTIME] = time.time() self[api.TXN_ID] = bll_request[api.TXN_ID] self[api.DATA] = [] # short cuts self.txn_id = self.get(api.TXN_ID) def complete(self, state=None): self[api.ENDTIME] = time.time() self[api.DURATION] = self[api.ENDTIME] - self[api.STARTTIME] current_state = self.get(api.STATUS) if state: # If a state is supplied use it self[api.STATUS] = state elif current_state is None or current_state == api.STATUS_INPROGRESS: self[api.STATUS] = api.COMPLETE # Otherwise leave self[api.STATUS] unchanged def error(self, cause): self[api.DATA] = [] # make sure data is a list before appending self[api.STATUS] = api.STATUS_ERROR self[api.DATA].append({api.DATA: cause}) return self def exception(self, stack_trace): self[api.DATA] = [] # make sure data is a list before appending self[api.STATUS] = api.STATUS_ERROR self[api.DATA].append({'stack_trace': stack_trace}) return self def __str__(self): return response_to_string(self) 070701000DFF81000081A40000000000000000000000015B64B4DF000002FE000000FD0000000200000000000000000000003F00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/bll/app.py# (c) Copyright 2015-2017 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC import atexit import logging from pecan import make_app from bll.common.util import setup_txn_logging LOG = logging.getLogger(__name__) def setup_app(config): app_conf = dict(config.app) app = make_app( app_conf.pop('root'), logging=getattr(config, 'logging', {}), **app_conf ) setup_txn_logging() LOG.info('*** BLL service started ****') return app @atexit.register def app_exit(): # pragma: no coverage """ Upon Teardown of application, report shutdown. """ # In unit test environments, _logger_ may be undefined at this point if LOG: LOG.info('*** BLL service stopped ****') 070701000DFFAF000041ED0000000000000000000000025B64B4DF00000000000000FD0000000200000000000000000000003F00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/bll/common070701000DFFB1000081A40000000000000000000000015B64B4DF00000000000000FD0000000200000000000000000000004B00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/bll/common/__init__.py070701000DFFB2000081A40000000000000000000000015B64B4DF0000030D000000FD0000000200000000000000000000004C00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/bll/common/exception.py# (c) Copyright 2015-2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC def _(msg): """ Define _ to just return its arg. Using this will flag strings that call it for translation. The strings are actually localized in service.py before being returned to the UI """ return msg class BllException(Exception): """ This class and its derived classes should define an ``overview`` member that contains a string describing an overview of the type of error. """ overview = _("An unknown exception occurred") class InvalidBllRequestException(BllException): overview = _("Invalid request") class BllAuthenticationFailedException(BllException): overview = _("Authentication Failed to Backend Identity") 070701000DFFB6000081A40000000000000000000000015B64B4DF000002F3000000FD0000000200000000000000000000004700000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/bll/common/i18n.py# (c) Copyright 2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC import gettext import os def get_(language): """ Returns the function for localizing text for the given language, which is normally assigned to the function name ``_``. Typical usage:: _ = get_('en') print _("localizable string") :param language: :return: """ locale_dir = os.path.realpath(os.path.join( os.path.dirname(__file__), '..', 'locale')) translator = gettext.translation(domain='messages', localedir=locale_dir, fallback=True, languages=[language]) return translator.ugettext 070701000DFFB5000081A40000000000000000000000015B64B4DF00000BFA000000FD0000000200000000000000000000004D00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/bll/common/job_status.py# (c) Copyright 2016-2017 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC from datetime import datetime, timedelta import json import logging import pecan import pymysql.cursors from bll import api LOG = logging.getLogger(__name__) def _get_status_obj(): # This function exists merely to facilitate injecting a test double # during tests where mysql is not available return DbStatus() def update_job_status(txn_id, status): return _get_status_obj().update_job_status(txn_id, status) def get_job_status(txn_id): return _get_status_obj().get_job_status(txn_id) class DbStatus(object): """ Implementation using a mysql database. This is suitable for production clustered environments """ def _get_connection(self): # Retrieve a database connection based on the config config = pecan.conf.db.to_dict() config['cursorclass'] = pymysql.cursors.DictCursor connection = pymysql.connect(**config) return connection def update_job_status(self, txn_id, status): connection = self._get_connection() try: with connection.cursor() as cursor: sql = "SELECT `status` FROM `jobs` WHERE `id`=%s" cursor.execute(sql, txn_id) row = cursor.fetchone() timestamp = datetime.now() message = "" try: message = json.dumps(status) except TypeError: message = json.dumps(str(status)) if row is None: sql = """ INSERT INTO `jobs` (`id`, `updated_at`, `status`) VALUES (%s, %s, %s) """ parms = [txn_id, timestamp, message] else: sql = """ UPDATE `jobs` SET `updated_at`=%s, `status`=%s WHERE `id`=%s """ parms = [timestamp, message, txn_id] cursor.execute(sql, parms) connection.commit() dayold = timestamp - timedelta(days=1) with connection.cursor() as cursor: sql = "DELETE FROM `jobs` where `updated_at` < %s" cursor.execute(sql, dayold) connection.commit() except Exception as e: LOG.exception(e) finally: connection.close() def get_job_status(self, txn_id): connection = self._get_connection() try: with connection.cursor() as cursor: sql = "SELECT `status` FROM `jobs` WHERE `id`=%s" cursor.execute(sql, txn_id) row = cursor.fetchone() if row is None: return {api.STATUS: api.STATUS_NOT_FOUND} return json.loads(row.get("status")) except Exception as e: LOG.exception(e) finally: connection.close() 070701000DFFB0000081A40000000000000000000000015B64B4DF00002323000000FD0000000200000000000000000000004700000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/bll/common/util.py# (c) Copyright 2015-2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017-2018 SUSE LLC from dogpile.cache.api import CachedValue import re import logging import threading import time from bll import api from uuid import uuid4, uuid1 from pecan import conf # the super-secret recipe for searching key-values pairs (i.e. key: value) RE_KEYVALUE = re.compile( r'(\\?[\'"])(.*?)(\\?[\'"]):\s*(\\?[\'"])(.*?)(\\?[\'"])([:\s,}\]])') context = threading.local() def deepMerge(d1, d2, inconflict=lambda v1, v2: v2): ''' merge d2 into d1. using inconflict function to resolve the leaf conflicts ''' for k in d2: if k in d1: if isinstance(d1[k], dict) and isinstance(d2[k], dict): deepMerge(d1[k], d2[k], inconflict) elif d1[k] != d2[k]: d1[k] = inconflict(d1[k], d2[k]) else: d1[k] = d2[k] return d1 def scrub_passwords(_val): if _val is None: return _val try: # We want to get a string value out of this object # strings are immutable so _val is never modified. if type(_val) in (str, unicode): out = _val else: out = str(_val) # convert u'blah' to "blah" since 'u' literals are not json-compatible # but also convert non-unicode strings, too out = re.sub("u?'(.*?)'([:\s,}\]])", lambda match: '"%s"%s' % (match.group(1), match.group(2)), out) # Some Python-represented values may be None, but None is not a valid # value in json. Let's convert that None to "None". This isn't # necessary, but it makes it easier for unit-testing to take place out = re.sub(": None", ": \"None\"", out) # Search for key-value pairs where the key contains 'password/cert' # and "4-star" the values out = re.sub(RE_KEYVALUE, lambda match: "%s%s%s: %s%s%s%s" % (match.group(1), match.group(2), match.group(3), match.group(4), '****', match.group(6), match.group(7)) if 'password' in match.group(2).lower() or 'cert' in match.group(2).lower() else match.group(0), out) # Search for key-value pairs where the key contains 'token' out = re.sub(RE_KEYVALUE, lambda match: "%s%s%s: %s%s%s%s" % (match.group(1), match.group(2), match.group(3), match.group(4), "%s%s" % ('*' * (len(match.group(5)) - 4), match.group(5) [-(min(4, len(match.group(5)))):]), match.group(6), match.group(7)) if 'token' in match.group(2).lower() else match.group(0), out) # we only need to return a string representation of the object # since this is only used for logging return out except Exception: # Not sure how we'd ever get here, but just in case we can't scrub # the passwords, we'll just print this useless message return "Could not properly scrub_passwords for passed in object" def empty(value): if value is None: return True elif isinstance(value, str) and len(value) < 1: return True return False def get_conf(key, default=None): """ Traverse through the levels of the config file to obtain the specified key, safely handling any missing levels. For example, if the key is "app.error", it will find the "error" entry in the "app" dictionary; if either the app dictionary or the error entry is missing, then the default will be returned. """ return get_val(conf, key, default) def get_val(dic, key, default=None): """ Traverse through the levels of a dictionary to obtain the specified key, safely handling any missing levels. For example, if the key is "app.error", it will find the "error" entry in the "app" dictionary; if either the app dictionary or the error entry is missing, then the default will be returned. """ if not key: return default try: current = dic for attribute in key.split("."): current = current[attribute] return current except KeyError: return default def setup_txn_logging(): """ Setup python logging to be able to include the txn id in messages, which will be kept in thread-local storage """ LOG = logging.getLogger('bll') f = ContextFilter() LOG.addFilter(f) for handler in LOG.handlers: handler.addFilter(f) class ContextFilter(logging.Filter): """ This logging filter enables the transaction id to be included as a field in log messages """ def filter(self, record): record.txn_id = getattr(context, 'txn_id', '') return True def new_txn_id(txn_id=None): """ Create a new txn id using an exiting transaction id. The newly created txn id will contain a concatenation of the existing transaction id plus a new unique id, separated by a delimiter. The basic idea is that when a new call comes into the BLL, it has a transaction id that is often a uuid. If the BLL first service needs to make another call to another BLL service, it is required that it have a new transaction id since transaction id's are used as unique identifiers for obtaining status of long-running processes, which must be unique or else it cannot properly correlate calls with responses. Creating transaction ids for these subsequent calls that are derived from the original facilitate associating the calls together. :param txn_id: :return: """ main_txn_id = txn_id # No txn_id passed, so look in the context if not main_txn_id: main_txn_id = getattr(context, 'txn_id', '') if main_txn_id: # If we have a txn_id, then create a sub-txn_id with a unique id. # This sub-txn does not have to be a full uuid -- instead, generate # a uuid1 whose first 8 characters are unique on this system, # and thus for this transaction (the 8 characters are generated by a # timestamp plus logic to avoid duplicates) return main_txn_id.split('.')[0] + "." + str(uuid1())[:8] else: return str(uuid4()) def get_service_tenant_name(): return get_conf('keystone.service_tenant', 'service') def response_to_string(resp, print_data=False): """ Utility function for converting a response dictionary to a string. This function can be used to create a string representation of a BllResponse, which is a thin veneer over a dictionary, or just a bare dictionary (which is all that the app_controller has) :param resp: :return string: """ # Print the DATA portion of the request with sorted keys if print_data: data = resp.get(api.DATA) if isinstance(data, dict): sorted_keys = sorted(data.keys()) data_list = [] for key in sorted_keys: data_list.append("%s:%s" % (key, str(data[key]))) data_str = scrub_passwords("{%s}" % ",".join(data_list)) else: data_str = scrub_passwords(data) data_str = "DATA:%s" % data_str else: data_str = "" result = "" if api.STATUS in resp: result += "STATUS:%s " % (resp[api.STATUS]) if api.TXN_ID in resp: result += "TXN:%s " % (resp[api.TXN_ID]) if api.DURATION in resp: result += "DURATION:%d " % (resp[api.DURATION]) if api.PROGRESS in resp: result += "PROGRESS:%s " % (str(resp[api.PROGRESS])) return result + data_str def start_cache_cleaner(cache, cache_expiration, thread_name="CacheCleaner"): """ Create a thread to clean up the given dogpile cache. Items whose age, in seconds, is older than cache_expiration, are removed from the cache. """ def expire_cache(): while True: current_time = time.time() for k, v in cache.items(): if isinstance(v, CachedValue): # Determines whether the cache has expired by comparing # the elapsed time since the creation time (metadata["ct"]) # against the expiration time. if current_time - v.metadata["ct"] > cache_expiration: del cache[k] time.sleep(cache_expiration) t = threading.Thread(target=expire_cache, name=thread_name) t.daemon = True t.start() 070701000DFFAE000081A40000000000000000000000015B64B4DF0000067D000000FD0000000200000000000000000000004100000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/bll/hooks.py# (c) Copyright 2015-2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC import logging from pecan.hooks import PecanHook LOG = logging.getLogger(__name__) class RestHook(PecanHook): ''' RestHook is class used to log info before() and after() controller code is run. If an exception is raised in the controller then this is handled by on_error(). ''' def before(self, state): ''' Using 'before' hook to inspect state before controller code is run. ''' LOG.info('Request from: %s "%s %s"' % (state.request.remote_addr, state.request.method, state.request.path_qs)) def after(self, state): ''' Using 'after' hook to inspect state after controller code is run. ''' LOG.info('Response to: %s "%s %s" %s %s' % (state.request.remote_addr, state.request.method, state.request.path_qs, state.response.status_code, state.response.content_length)) def on_error(self, state, e): ''' Using 'on_error' hook to inspect state when exception raised in controller. Will have default status_code of "Bad Request", 400 ''' status_code = getattr(e, 'status_code', 400) state.response.status = state.response.status_code = status_code LOG.warn('Exception %r in: %s "%s %s" %s %s' % (e, state.request.remote_addr, state.request.method, state.request.path_qs, state.response.status_code, state.response.content_length)) return state.response 070701000DFFA0000041ED0000000000000000000000055B64B4DF00000000000000FD0000000200000000000000000000003F00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/bll/locale070701000DFFA6000041ED0000000000000000000000035B64B4DF00000000000000FD0000000200000000000000000000004200000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/bll/locale/en070701000DFFA7000041ED0000000000000000000000025B64B4DF00000000000000FD0000000200000000000000000000004E00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/bll/locale/en/LC_MESSAGES070701000DFFA8000081A40000000000000000000000015B64B4DF0000081A000000FD0000000200000000000000000000005A00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/bll/locale/en/LC_MESSAGES/messages.po# English translations for bll. # (c) Copyright 2016-2017 Hewlett Packard Enterprise Development LP # (c) Copyright 2017-2018 SUSE LLC # This file is distributed under the same license as the bll # project. # Gary Smith, 2016. # #, fuzzy msgid "" msgstr "" "Project-Id-Version: bll 1.0\n" "Report-Msgid-Bugs-To: EMAIL@ADDRESS\n" "POT-Creation-Date: 2018-01-16 08:43-0800\n" "PO-Revision-Date: 2016-10-05 21:06-0700\n" "Last-Translator: FULL NAME \n" "Language: en\n" "Language-Team: en \n" "Plural-Forms: nplurals=2; plural=(n != 1)\n" "MIME-Version: 1.0\n" "Content-Type: text/plain; charset=utf-8\n" "Content-Transfer-Encoding: 8bit\n" "Generated-By: Babel 2.5.1\n" #: bll/common/exception.py:19 msgid "An unknown exception occurred" msgstr "" #: bll/common/exception.py:23 msgid "Invalid request" msgstr "" #: bll/common/exception.py:27 msgid "Authentication Failed to Backend Identity" msgstr "" #: bll/plugins/ardana_service.py:181 msgid "Unknown error occurred, please refer to the following ardana log: {}" msgstr "" #: bll/plugins/monitor_service.py:400 msgid "" "cannot filter on dimension_name_filter: {} without a group_by on " "dimension_name" msgstr "" #: bll/plugins/nova_service.py:447 msgid "Could not find service to delete" msgstr "" #: bll/plugins/nova_service.py:449 msgid "Executed nova service delete on service {}" msgstr "" #: bll/plugins/nova_service.py:489 msgid "instance {} deleted" msgstr "" #: bll/plugins/preferences_service.py:53 msgid "User {} does not exist" msgstr "" #: bll/plugins/preferences_service.py:81 msgid "Cannot update non-existent user {}" msgstr "" #: bll/plugins/service.py:169 msgid "{0}: {1}" msgstr "" #: bll/plugins/service.py:184 msgid "Operation and action missing" msgstr "" #: bll/plugins/service.py:190 msgid "Unsupported operation: {}" msgstr "" #: bll/plugins/service.py:458 msgid "Timed out waiting for {}" msgstr "" #: bll/plugins/service.py:476 msgid "Failure calling {} service" msgstr "" #: bll/plugins/user_group_service.py:86 msgid "Invalid project: {}" msgstr "" 070701000DFFA9000041ED0000000000000000000000035B64B4DF00000000000000FD0000000200000000000000000000004200000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/bll/locale/ja070701000DFFAA000041ED0000000000000000000000025B64B4DF00000000000000FD0000000200000000000000000000004E00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/bll/locale/ja/LC_MESSAGES070701000DFFAB000081A40000000000000000000000015B64B4DF00000813000000FD0000000200000000000000000000005A00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/bll/locale/ja/LC_MESSAGES/messages.po# Japanese translations for bll. # (c) Copyright 2016-2017 Hewlett Packard Enterprise Development LP # (c) Copyright 2017-2018 SUSE LLC # This file is distributed under the same license as the bll # project. # Gary Smith, 2016. # #, fuzzy msgid "" msgstr "" "Project-Id-Version: bll 1.0\n" "Report-Msgid-Bugs-To: EMAIL@ADDRESS\n" "POT-Creation-Date: 2018-01-16 08:43-0800\n" "PO-Revision-Date: 2016-10-06 12:46-0700\n" "Last-Translator: FULL NAME \n" "Language: ja\n" "Language-Team: ja \n" "Plural-Forms: nplurals=1; plural=0\n" "MIME-Version: 1.0\n" "Content-Type: text/plain; charset=utf-8\n" "Content-Transfer-Encoding: 8bit\n" "Generated-By: Babel 2.5.1\n" #: bll/common/exception.py:19 msgid "An unknown exception occurred" msgstr "" #: bll/common/exception.py:23 msgid "Invalid request" msgstr "" #: bll/common/exception.py:27 msgid "Authentication Failed to Backend Identity" msgstr "" #: bll/plugins/ardana_service.py:181 msgid "Unknown error occurred, please refer to the following ardana log: {}" msgstr "" #: bll/plugins/monitor_service.py:400 msgid "" "cannot filter on dimension_name_filter: {} without a group_by on " "dimension_name" msgstr "" #: bll/plugins/nova_service.py:447 msgid "Could not find service to delete" msgstr "" #: bll/plugins/nova_service.py:449 msgid "Executed nova service delete on service {}" msgstr "" #: bll/plugins/nova_service.py:489 msgid "instance {} deleted" msgstr "" #: bll/plugins/preferences_service.py:53 msgid "User {} does not exist" msgstr "" #: bll/plugins/preferences_service.py:81 msgid "Cannot update non-existent user {}" msgstr "" #: bll/plugins/service.py:169 msgid "{0}: {1}" msgstr "" #: bll/plugins/service.py:184 msgid "Operation and action missing" msgstr "" #: bll/plugins/service.py:190 msgid "Unsupported operation: {}" msgstr "" #: bll/plugins/service.py:458 msgid "Timed out waiting for {}" msgstr "" #: bll/plugins/service.py:476 msgid "Failure calling {} service" msgstr "" #: bll/plugins/user_group_service.py:86 msgid "Invalid project: {}" msgstr "" 070701000DFFA1000081A40000000000000000000000015B64B4DF000007D6000000FD0000000200000000000000000000004C00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/bll/locale/messages.pot# Translations template for bll # (c) Copyright 2016-2017 Hewlett Packard Enterprise Development LP # (c) Copyright 2017-2018 SUSE LLC # This file is distributed under the same license as the bll # Gary Smith, 2016. # #, fuzzy msgid "" msgstr "" "Project-Id-Version: bll 1.0\n" "Report-Msgid-Bugs-To: EMAIL@ADDRESS\n" "POT-Creation-Date: 2018-01-16 08:43-0800\n" "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n" "Last-Translator: FULL NAME \n" "Language-Team: LANGUAGE \n" "MIME-Version: 1.0\n" "Content-Type: text/plain; charset=utf-8\n" "Content-Transfer-Encoding: 8bit\n" "Generated-By: Babel 2.5.1\n" #: bll/common/exception.py:19 msgid "An unknown exception occurred" msgstr "" #: bll/common/exception.py:23 msgid "Invalid request" msgstr "" #: bll/common/exception.py:27 msgid "Authentication Failed to Backend Identity" msgstr "" #: bll/plugins/ardana_service.py:181 msgid "Unknown error occurred, please refer to the following ardana log: {}" msgstr "" #: bll/plugins/monitor_service.py:400 msgid "" "cannot filter on dimension_name_filter: {} without a group_by on " "dimension_name" msgstr "" #: bll/plugins/nova_service.py:447 msgid "Could not find service to delete" msgstr "" #: bll/plugins/nova_service.py:449 msgid "Executed nova service delete on service {}" msgstr "" #: bll/plugins/nova_service.py:489 msgid "instance {} deleted" msgstr "" #: bll/plugins/preferences_service.py:53 msgid "User {} does not exist" msgstr "" #: bll/plugins/preferences_service.py:81 msgid "Cannot update non-existent user {}" msgstr "" #: bll/plugins/service.py:169 msgid "{0}: {1}" msgstr "" #: bll/plugins/service.py:184 msgid "Operation and action missing" msgstr "" #: bll/plugins/service.py:190 msgid "Unsupported operation: {}" msgstr "" #: bll/plugins/service.py:458 msgid "Timed out waiting for {}" msgstr "" #: bll/plugins/service.py:476 msgid "Failure calling {} service" msgstr "" #: bll/plugins/user_group_service.py:86 msgid "Invalid project: {}" msgstr "" 070701000DFFA3000041ED0000000000000000000000035B64B4DF00000000000000FD0000000200000000000000000000004200000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/bll/locale/zh070701000DFFA4000041ED0000000000000000000000025B64B4DF00000000000000FD0000000200000000000000000000004E00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/bll/locale/zh/LC_MESSAGES070701000DFFA5000081A40000000000000000000000015B64B4DF00000819000000FD0000000200000000000000000000005A00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/bll/locale/zh/LC_MESSAGES/messages.po# Chinese translations for bll. # (c) Copyright 2016-2017 Hewlett Packard Enterprise Development LP # (c) Copyright 2017-2018 SUSE LLC # This file is distributed under the same license as the bll # project. # Gary Smith, 2016. # #, fuzzy msgid "" msgstr "" "Project-Id-Version: bll 1.0\n" "Report-Msgid-Bugs-To: EMAIL@ADDRESS\n" "POT-Creation-Date: 2018-01-16 08:43-0800\n" "PO-Revision-Date: 2016-10-05 21:06-0700\n" "Last-Translator: FULL NAME \n" "Language: zh\n" "Language-Team: zh \n" "Plural-Forms: nplurals=2; plural=(n != 1)\n" "MIME-Version: 1.0\n" "Content-Type: text/plain; charset=utf-8\n" "Content-Transfer-Encoding: 8bit\n" "Generated-By: Babel 2.5.1\n" #: bll/common/exception.py:19 msgid "An unknown exception occurred" msgstr "" #: bll/common/exception.py:23 msgid "Invalid request" msgstr "" #: bll/common/exception.py:27 msgid "Authentication Failed to Backend Identity" msgstr "" #: bll/plugins/ardana_service.py:181 msgid "Unknown error occurred, please refer to the following ardana log: {}" msgstr "" #: bll/plugins/monitor_service.py:400 msgid "" "cannot filter on dimension_name_filter: {} without a group_by on " "dimension_name" msgstr "" #: bll/plugins/nova_service.py:447 msgid "Could not find service to delete" msgstr "" #: bll/plugins/nova_service.py:449 msgid "Executed nova service delete on service {}" msgstr "" #: bll/plugins/nova_service.py:489 msgid "instance {} deleted" msgstr "" #: bll/plugins/preferences_service.py:53 msgid "User {} does not exist" msgstr "" #: bll/plugins/preferences_service.py:81 msgid "Cannot update non-existent user {}" msgstr "" #: bll/plugins/service.py:169 msgid "{0}: {1}" msgstr "" #: bll/plugins/service.py:184 msgid "Operation and action missing" msgstr "" #: bll/plugins/service.py:190 msgid "Unsupported operation: {}" msgstr "" #: bll/plugins/service.py:458 msgid "Timed out waiting for {}" msgstr "" #: bll/plugins/service.py:476 msgid "Failure calling {} service" msgstr "" #: bll/plugins/user_group_service.py:86 msgid "Invalid project: {}" msgstr "" 070701000DFF82000041ED0000000000000000000000025B64B4DF00000000000000FD0000000200000000000000000000004000000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/bll/plugins070701000DFF8B000081A40000000000000000000000015B64B4DF00000000000000FD0000000200000000000000000000004C00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/bll/plugins/__init__.py070701000DFF87000081A40000000000000000000000015B64B4DF00001E2D000000FD0000000200000000000000000000005200000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/bll/plugins/ardana_service.py# (c) Copyright 2016-2017 Hewlett Packard Enterprise Development LP # (c) Copyright 2017-2018 SUSE LLC from bll import api from bll.common.util import get_conf from bll.plugins.service import expose, SvcBase from bll.common.exception import InvalidBllRequestException import logging import requests import json import time LOG = logging.getLogger(__name__) class ArdSvc(SvcBase): """ The ardana service is written does not have a python client, since it is written in JavaScript and runs under node. Therefore all interaction is done via direct REST calls The ``target`` value for this plugin is ``ardana``. See :ref:`rest-api` for a full description of the request and response formats. """ TASK_POLL_INTERVAL = 10 def __init__(self, *args, **kwargs): super(ArdSvc, self).__init__(*args, **kwargs) url = self.token_helper.get_service_endpoint('lifecycle') self.base_url = "/".join([url.strip(), 'api/v2']) if not self.operation: self.operation = 'do_path_operation' @expose() def delete_compute_host(self): """ Delete a host Once the delete operation successfully completes ensure that nova-service delete call is made. This will be removed as soon as the delete host playbook is created, (which will be called from the ardana backend) Request format:: "target": "ardana", "operation": "delete_compute_host", "data": { "request_data": { "serverid": "MYID", "novaServiceDelete": { "hostname": "MYHOSTNAME" }, ... } } """ # Ensure we have the required properties to start with request_data = self.data[api.REQUEST_DATA] serverid = request_data['serverid'] hostname = request_data['novaServiceDelete']['hostname'] # Make the usual ardana delete request to /server url = 'servers/%s/process' % serverid self._request(url, action='DELETE') # Successfully removed the server from the model and updated config # processor output. Need now to ensure compute host is deleted from # the nova world return self.call_service(target="nova", operation="service-delete", data={'hostname': hostname}) @expose() def do_path_operation(self): """ Perform an operation against the given path in ardana_service. Example payload (``request_data`` and ``request_parameters`` sections are optional):: "target": "ardana", "operation": "do_path_operation", "data": { "path": "model/entities/server", "request_data": { # OPTIONAL "limit": "MYLIMIT", "encryptionKey": "MYKEY" }, "request_parameters": [ "MYKEY=MYVALUE", ...] # OPTIONAL } """ # Convert list of key=value strings into a dictionary, .e.g. # [ "key1=value1", "key2=value2" ] => # { 'key1': "value1", "key2": "value2" } query_parms = None if api.REQUEST_PARAMETERS in self.data: parms_list = self.data[api.REQUEST_PARAMETERS] query_parms = dict([x.split('=') for x in parms_list]) return self._request(self.data[api.PATH], query_parms, self.data.get(api.REQUEST_DATA), action=self.action or 'GET') @expose() def get_network_data(self): """ Aggregates the networking information in the servers list to return a list of networks across all servers. Example payload: "target": "ardana", "operation": "get_network_data" """ server_info = self._request('model/cp_output/server_info.yml') agg_networks = {} for host_data in server_info.values(): for networks in host_data['net_data'].values(): for network_name in networks: if network_name not in agg_networks: agg_networks[network_name] = networks[network_name] # optionally, remove host-specific/unimportant data from each network for network_data in agg_networks.values(): for unwanted_key in ['addr', 'endpoints']: network_data.pop(unwanted_key, None) return agg_networks.values() @expose(is_long=True) def run_playbook(self, validate): """ Performs an ansible playbook operation. Request format:: "target": "ardana", "operation": "run_playbook", "data": { "playbook_name": "MYPLAYBOOK", # REQUIRED "playbook_PARAM1": "MYVALUE", # OPTIONAL ... } """ if validate: playbook_name = self.data.pop('playbook_name', None) if not playbook_name: raise InvalidBllRequestException('unspecified playbook ' 'to run') # Initiate the request and get its reference req_path = 'playbooks/' + playbook_name resp = self._request(req_path, body=self.data, action='POST') self.ref_id = resp['id'] self.status_path = 'plays/' + self.ref_id self.update_job_status(resp, 25) return else: still_alive = True try: while still_alive: poll_resp = self._request(self.status_path) still_alive = poll_resp.get('alive', False) if still_alive: self.update_job_status(poll_resp, 50) time.sleep(self.TASK_POLL_INTERVAL) # We have no idea how long a playbook is going to take return poll_resp except Exception as e: LOG.exception(e) self.response.error(self._( 'Unknown error occurred, please refer to the following ' 'ardana log: {}').format(self.ref_id)) return self.response def _request(self, relative_path, query_params=None, body=None, action='GET'): url = "/".join([self.base_url, relative_path.strip('/')]) if isinstance(body, dict): body = json.dumps(body) headers = { 'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': self.token, 'User-Agent': api.USER_AGENT } response = requests.request(action, url, params=query_params, data=body, headers=headers, verify=not get_conf("insecure")) if 400 <= response.status_code < 600: # Raise an exception if not found. The content has the error # message to return try: message = response.json()['message'] except Exception: message = response.content raise Exception(message) return response.json() @classmethod def needs_services(cls): return ['ardana'] 070701000DFF86000081A40000000000000000000000015B64B4DF0000296F000000FD0000000200000000000000000000005300000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/bll/plugins/catalog_service.py# (c) Copyright 2016-2017 Hewlett Packard Enterprise Development LP # (c) Copyright 2017-2018 SUSE LLC import logging import stevedore from keystoneclient.v3 import client as ksclient from bll import api from bll.plugins import service from bll.common.util import get_conf, get_val LOG = logging.getLogger(__name__) def on_load_failures(manager, entrypoint, exception): """ Some plugins import modules that may not be present in some environments, which is normally not an error. """ if isinstance(exception, ImportError): LOG.warn("Error loading %s: %s", entrypoint.module_name, exception) else: LOG.exception(exception) class CatalogSvc(service.SvcBase): """ Obtain a catalog of BLL plugins and openstack services available in the current environment. The ``target`` value for this plugin is ``catalog``. See :ref:`rest-api` for a full description of the request and response formats. """ @service.expose() def get_plugins(self): """ Gets the list of BLL plugin names whose dependent services are available. If monasca-transform information is present in the config file, then ``monasca-transform`` will also be returned. Request format:: "target": "catalog", "operation": "get_plugins" :return: List of BLL plugin names. For example:: [ "ace", "catalog", "monasca-transform", "monitor" ] """ available = self._get_services() mgr = stevedore.extension.ExtensionManager( 'bll.plugins', on_load_failure_callback=on_load_failures) plugins = [] for ext in mgr: if ext.plugin.is_available(available): plugins.append(ext.name) # monasca-transform components will not show up in keystone's list of # services, so search for it's existence in config's services if get_conf('services.monasca.components.monasca-transform'): plugins.append('monasca-transform') return sorted(plugins) @service.expose() def get_services(self): """ Gets the list of services reported by keystone. The list will contain both service names and service types. e.g, both ``identity`` and ``keystone`` will be returned for keystone. Request format:: "target": "catalog", "operation": "get_services" :return: List of services reported by keystone. For example:: [ "glance", "identity", "image", "keystone" ] """ return sorted(self._get_services()) @service.expose() def get_compute_clusters(self): """ Returns compute cluster data with control plane and region data. Refer to an example of services in Ardana's service_topology.yml Request format:: "target": "catalog", "operation": "get_compute_clusters" :return: Cluster info for all compute hosts. """ nc = self._get_service_topo('services.nova.components') # We need to filter out non-existent compute hosts (i.e. baremetal) if 'hypervisor-list' in self.data: hyp_list = self.data['hypervisor-list'] else: hyp_list = self.call_service(target='nova', operation='hypervisor-list', data={'include_status': False}) old_cplanes = nc['nova-compute']['control_planes'] new_cplanes = {} for hyp in hyp_list: # Use the hypervisor's service_host, which appears to match what is # in the model and works with kvm hosts, ch_name = hyp['service_host'] ch_region = hyp['region'] # go thru each control plane and see if the host and region from # the hypervisor list exists. If so, add this into new_cplanes for cp_name, cp in old_cplanes.iteritems(): if ch_region not in cp['regions']: continue for type in ('resources', 'clusters'): if type not in cp: continue for cl_name, cl_hosts in cp[type].iteritems(): if ch_name not in cl_hosts: continue new_cplanes = self._insert_cluster_data( new_cplanes, cp_name, cp['regions'], type, cl_name, ch_name ) nc['nova-compute']['control_planes'] = new_cplanes return self._derive_clusters(nc['nova-compute']) def _insert_cluster_data(self, cplanes, cp_name, regions, cl_type, cl_name, comp_hostname): """ This method adds the given comp_hostname into the appropriate area of the cplanes (control planes) structure, which would look something like this: cplanes = { cp_name: { 'regions': regions, cl_type: { cl_name: [, comp_hostname] } } } """ if cp_name not in cplanes: cplanes[cp_name] = {} cp = cplanes[cp_name] cp['regions'] = regions if cl_type not in cp: cp[cl_type] = {} clusters = cp[cl_type] if cl_name not in clusters: clusters[cl_name] = [] cluster = clusters[cl_name] if comp_hostname not in cluster: cluster.append(comp_hostname) return cplanes @service.expose() def get_swift_clusters(self): """ Returns swift cluster data with control plane and region data. example of services in Ardana's service_topology.yml Request format:: "target": "catalog", "operation": "get_swift_clusters" :return: Cluster info for all swift clusters """ sc = self._get_service_topo('services.swift.components') return self._derive_clusters(get_val(sc, 'swift-account', {}), get_val(sc, 'swift-container', {}), get_val(sc, 'swift-object', {}), get_val(sc, 'swift-proxy', {})) def _get_service_topo(self, path): """ Gets the requested section from service_topology.yml in the config file and if it doesn't succeed, try to get it from ardana service """ # First try getting it from the config file (blazingly fast) comp_topo = get_conf(path) if comp_topo: return comp_topo else: # Otherwise, try getting it from ardana service (pretty slow) services = self.call_service( target='ardana', operation='do_path_operation', data={'path': '/model/cp_output/service_topology.yml'} ) for key in path.split('.'): services = services.get(key, {}) return services def _derive_clusters(self, *comps): """ For each component in *comps, return a dict of clusters Refer to an example of services in Ardana's service_topology.yml """ result = {} # for each component for comp in comps: control_planes = comp.get('control_planes', []) # for each control plane for cp in control_planes: for cluster_type in ('resources', 'clusters'): clusters = control_planes[cp].get(cluster_type, []) # for each cluster/resource for cl in clusters: # cluster names = # ":" clust_name = "%s:%s" % (cp, cl) # Concatenate the hosts to the cluster's list of hosts host_list = clusters[cl] + result.get(clust_name, []) # keep the cluster's list of hosts unique and sorted result[clust_name] = sorted(list(set(host_list))) return result def _get_services(self): client = ksclient.Client(session=self.token_helper.get_session(), endpoint_type=get_conf( "services.endpoint_type", default="internalURL"), interface=get_conf("services.interface", default="internal"), user_agent=api.USER_AGENT, verify=not get_conf("insecure")) services = [] for svc in client.services.list(): services.append(svc.name) services.append(svc.type) return services @service.expose() def get_regions(self): """ Obtain a list of regions available in the current environment. Request format:: "target": "catalog", "operation": "get_regions" """ regions = [] for region in self.token_helper.get_regions(): regions.append({'id': region.id, 'description': region.description or region.id}) return regions @service.expose('get_enterprise_app_endpoints') def get_enterprise_app_endpoints(self): """ Returns a list of endpoints (URLs) of enterprise applications: oo, csa, mpp, Request format:: "target": "catalog", "operation": "get_enterprise_app_endpoints" """ response = {} endpoints = [ {'service_type': 'enterprise-oo', 'result': 'oo'}, {'service_type': 'enterprise-csa', 'result': 'csa'}, {'service_type': 'enterprise-mpp', 'result': 'mpp'}, {'service_type': 'dashboard', 'result': 'horizon'}, ] for endpoint in endpoints: # Get public URLs since user will navigate there via a browser url = self.token_helper.get_service_endpoint( endpoint['service_type'], 'publicURL') response[endpoint['result']] = url return response 070701000DFF89000081A40000000000000000000000015B64B4DF00000C5E000000FD0000000200000000000000000000005200000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/bll/plugins/cinder_service.py# (c) Copyright 2015-2017 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC from cinderclient.v2 import client as cinderclient from bll import api from bll.common.util import get_conf from bll.plugins import service class CinderSvc(service.SvcBase): """ This class deals with all the interaction with cinder client including cinder volume type creation and mapping the volume backends to volume types. The ``target`` value for this plugin is ``cinder``. See :ref:`rest-api` for a full description of the request and response formats. """ def __init__(self, *args, **kwargs): """ Initializer for the Cinder Client Service """ super(CinderSvc, self).__init__(*args, **kwargs) self.cinder_client = cinderclient.Client( session=self.token_helper.get_session(), endpoint_type=get_conf("services.endpoint_type", default="internalURL"), user_agent=api.USER_AGENT) @service.expose(action="DELETE") def volume_type_delete(self): """ Delete a volume type. Request format:: "target": "cinder", "operation": "volume_type_delete", "action": "DELETE", "volume_type_id": "MYID" """ vol_type = self.request[api.DATA]["volume_type_id"] response = self.cinder_client.volume_types.delete(vol_type) return response @service.expose(action="PUT") def volume_type_add(self): """ Add a volume type with the given name. Request format:: "target": "cinder", "operation": "volume_type_add", "action": "PUT", "volume_type": "MYTYPENAME" """ vol_type = self.request[api.DATA]["volume_type"] volume_type = self.cinder_client.volume_types.create(vol_type) response = {'id': volume_type.id, 'name': volume_type.name} return response @service.expose() def volume_type_list(self): """ Return a list of volume types. Request format:: "target": "cinder", "operation": "volume_type_list" """ volume_types = self.cinder_client.volume_types.list() return {v.id: v.name for v in volume_types} @service.expose(action="PUT") def map_volume_backend(self): """ Updates the ``volume_backend_name`` extra specs of a volume type to refer to the given backend name. Request format:: "target": "cinder", "operation": "volume_type_list", "volume_type_id": "MYID", "backend_name": "MYBACKEND" """ vol_type_id = self.request[api.DATA]["volume_type_id"] backend_name = self.request[api.DATA]["backend_name"] response = {} volume_type = self.cinder_client.volume_types.get(vol_type_id) resp = volume_type.set_keys({"volume_backend_name": backend_name}) response[vol_type_id] = resp return response @classmethod def needs_services(cls): return ['volume'] 070701000DFF83000081A40000000000000000000000015B64B4DF00003D5F000000FD0000000200000000000000000000005300000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/bll/plugins/compute_service.py# (c) Copyright 2015-2017 Hewlett Packard Enterprise Development LP # (c) Copyright 2017-2018 SUSE LLC import logging from datetime import datetime, timedelta from bll import api from bll.plugins.monitor_service import TYPE_UNKNOWN from bll.plugins.service import SvcBase, expose LOG = logging.getLogger(__name__) # Generic utilization metrics from monasca to be passed back to the UI USED_CPU_PERC = 'used_cpu_perc' USED_MEMORY_MB = 'used_memory_mb' USED_STORAGE_MB = 'used_storage_mb' TOTAL_MEMORY_MB = 'total_memory_mb' TOTAL_STORAGE_MB = 'total_storage_mb' # This is not passed back to UI, but used to calculate USED_MEMORY_MB USABLE_MEMORY_MB = 'usable_memory_mb' MONASCA_DETAIL_DATA = {USED_CPU_PERC: -1, USED_MEMORY_MB: -1, USED_STORAGE_MB: -1, TOTAL_MEMORY_MB: -1, TOTAL_STORAGE_MB: -1, USABLE_MEMORY_MB: -1} METRICS_MAP = {'cpu.system_perc': USED_CPU_PERC, 'mem.total_mb': TOTAL_MEMORY_MB, 'disk.total_used_space_mb': USED_STORAGE_MB, 'disk.total_space_mb': TOTAL_STORAGE_MB, 'mem.usable_mb': USABLE_MEMORY_MB} # The possible VM types returned by nova are defined in # https://github.com/openstack/nova/blob/master/nova/compute/hv_type.py . # This mapping converts nova types into the ones that the UI expects TYPE_CLUSTER = 'cluster' TYPE_KVM = 'kvm' TYPE_HYPERV = 'hyperv' NOVA_TYPE_MAPPINGS = {"hyperv": TYPE_HYPERV, "kvm": TYPE_KVM, "qemu": TYPE_KVM, "vmware": TYPE_CLUSTER} NOVA_STATUS_MAPPING = {"enabled": "ok", "disabled": "error"} NOVA_STATE_MAPPING = {"up": "activated", "down": "deactivated"} NOVA_DETAIL_DATA = {'allocated_cpu': -1, 'allocated_memory': -1, 'allocated_storage': -1, 'cloud_external_interface': '', 'cloud_trunk': '', 'cloud_trunk_interface': '', 'hypervisor': '', 'hypervisor_display_name': '', 'hypervisor_hostname': '', 'hypervisor_id': "UNSET", 'id': '', 'instances': -1, 'name': '', 'ping_status': TYPE_UNKNOWN, 'progress': 0, 'region': '', 'service_host': '', 'state': '', 'status': TYPE_UNKNOWN, 'technology': '', 'total_cpu': -1, 'total_memory': -1, 'total_storage': -1, 'type': ''} HYPERVISOR_DISPLAYNAME_MAPPINGS = {TYPE_CLUSTER: 'ESXi', TYPE_KVM: 'KVM', TYPE_HYPERV: 'Hyper-V'} HYPERVISOR_MAPPINGS = {TYPE_CLUSTER: 'esx'} TECHNOLOGY_MAPPINGS = {TYPE_KVM: 'KVM', TYPE_CLUSTER: 'VMWARE', TYPE_HYPERV: 'Microsoft'} class ComputeSvc(SvcBase): """ The ``target`` value for this plugin is ``compute``. See :ref:`rest-api` for a full description of the request and response formats. """ @expose() def get_compute_list(self): """ Gets the list of compute hosts. The list of hypervisors is first retrieved from nova. Request format:: "target": "compute", "operation": "get_compute_list" """ if 'hypervisor-list' in self.data: nova_hyp_list = self.data['hypervisor-list'] else: nova_hyp_list = self.call_service(target='nova', operation='hypervisor-list', data={'include_status': True}) nova_compute_list = self.get_compute_data(nova_hyp_list, True) nova_compute_list = \ self._filter_out_hosts(nova_compute_list, 'type', 'ironic') return nova_compute_list @expose('details') def get_compute_details(self): """ Obtains details of the given compute host. Monasca, nova, and ardana_service when available and necessary in order to get these details. Request format:: "target": "compute", "operation": "details" "data" : { "data" : { "id" : "MYID", "type": "MYTYPE" }} """ datadata = self.data.get(api.DATA) id = datadata.get("id") type = datadata.get("type") return self._get_resource_details(type, id) @staticmethod def _filter_out_hosts(compute_list, type, value): new_compute_list = [] for hypervisor in compute_list: if hypervisor[type] != value: new_compute_list.append(hypervisor) return new_compute_list def _get_utilization(self, resource_type, **data): results = {} try: if resource_type in ["rhel", "hlinux", "kvm"]: meas_results = \ self._get_monasca_meas_value(METRICS_MAP, data) if meas_results.get(USABLE_MEMORY_MB) >= 0 and \ meas_results.get(TOTAL_MEMORY_MB) > 0: meas_results[USED_MEMORY_MB] = \ meas_results[TOTAL_MEMORY_MB] - \ meas_results.get(USABLE_MEMORY_MB) if meas_results: results.update(meas_results) return results except Exception as e: LOG.error(e.message) return None def _get_monasca_meas_value(self, metrics, dim): results = {} for metric_name in metrics: try: start_time = (datetime.utcnow() - timedelta( minutes=5)).isoformat() data = { 'operation': 'measurement_list', 'name': metric_name, 'start_time': start_time, 'merge_metrics': True } # Get potential monasca parms we might need to provide data.update(dim) meas_list = self.call_service( target='monitor', data=data, ) # There should only be one measurement list. # If not, it means our dimension filter is not strict enough if meas_list and len(meas_list) > 0: meas = meas_list[0] value_idx = meas['columns'].index('value') # get the latest(last) measurement val = meas['measurements'][-1][value_idx] if isinstance(val, int) or isinstance(val, float): results[metrics[metric_name]] = val else: results[metrics[metric_name]] = None except Exception as e: LOG.error("Error getting metric value for metric: " "%s Reason: %s", metric_name, e.message) results[metrics[metric_name]] = None return results def get_compute_data(self, hyp_list=None, include_status=True): # Request a list of all hypervisors from nova. if not hyp_list: hyp_list = \ self.call_service(target='nova', operation='hypervisor-list', data={'include_status': include_status}) compute_list = [] # Loop through nova_response and apply the relevant portions for hyp in hyp_list: compute_data = NOVA_DETAIL_DATA.copy() for k, v in hyp.iteritems(): if k in compute_data: compute_data[k] = v compute_data['id'] = compute_data['hypervisor_id'] compute_data['type'] = compute_data['type'].lower() if compute_data['type'] in NOVA_TYPE_MAPPINGS: compute_data['type'] = NOVA_TYPE_MAPPINGS[compute_data['type']] compute_data['status'] = NOVA_STATUS_MAPPING.get( compute_data['status'], "unknown") if compute_data['state'] in NOVA_STATE_MAPPING: compute_data['state'] = NOVA_STATE_MAPPING.get( compute_data['state']) compute_data['hypervisor'] = \ HYPERVISOR_MAPPINGS.get(compute_data['type'], compute_data['type']) compute_data['hypervisor_display_name'] = \ HYPERVISOR_DISPLAYNAME_MAPPINGS.get(compute_data['type'], compute_data['type']) compute_data['technology'] = \ TECHNOLOGY_MAPPINGS.get(compute_data['type'], '') compute_list.append(compute_data) return compute_list def _get_resource_details(self, hypervisor_type, hypervisor_id): details = {} # Monasca operates on dimensions, not hypervisor_id, so get hostname # and use it to determine utilization hyp_list = self.call_service( target='nova', operation='hypervisor-list' ) hyp_dict = {hyp['hypervisor_id']: hyp for hyp in hyp_list} hostname = hyp_dict[int(hypervisor_id)]['name'] details['instances'] = hyp_dict[int(hypervisor_id)]['instances'] dim = {'hostname': hostname} details['monasca'] = \ self._get_utilization(hypervisor_type, dimensions=dim) # Use the hostname to reverse-lookup the server name from the # config processor output's perspective server_info = self.call_service( target='ardana', path='/model/cp_output/server_info.yml' ) cp_host = None for host, info in server_info.iteritems(): if info['hostname'] == hostname: cp_host = host break if not cp_host: return details # Use the cp_host to get server-group and role-related data details['ardana'] = self.call_service( target='ardana', path="/model/entities/servers/" + cp_host ) return details @expose() def get_cluster_utilization(self): """ Gets cpu, memory, storage utilization for compute hosts, using information obtained from nova and monasca. Request format:: "target": "compute", "operation": "get_cluster_utilization" """ # hypervisor-list is used by many calls in this method. Cache it # for use by others later. hyp_list = self.call_service(target='nova', operation='hypervisor-list', data={'include_status': True}) self.data['hypervisor-list'] = hyp_list # first, get the compute clusters in the environment clusters = self.call_service( target='catalog', data={'operation': 'get_compute_clusters', 'hypervisor-list': hyp_list} ) ############################################################ # TODO: We also need to get rid of get_compute_data and just # use get_compute_list. They're very similar. ############################################################ compute_list = self.get_compute_data(hyp_list=hyp_list) compute_list = self._filter_out_hosts(compute_list, 'type', 'ironic') compute_list2 = self.get_compute_list() comp2_dict = {ch['service_host']: ch for ch in compute_list2 if 'service_host' in ch} compute_hosts = {host['hypervisor_hostname']: host for host in compute_list} # esx hosts show up as hypervisor_hostname in nova but and as # service_host in "Ardana names" in monasca. So we need this map to # translate the name back to a name that monasca recognizes esx_equiv_hosts = {host['service_host']: host['hypervisor_hostname'] for host in compute_list} for metric_name, ui_equiv in METRICS_MAP.iteritems(): # For each monasca metric, get all the measurements # across all hosts start_time = (datetime.utcnow() - timedelta(minutes=5)).isoformat() meas_list = self.call_service( target='monitor', data={ 'operation': 'measurement_list', 'name': metric_name, 'start_time': start_time, 'group_by': '*', 'merge_metrics': True } ) for meas in meas_list: hostname = meas['dimensions']['hostname'] if hostname not in compute_hosts: if hostname not in esx_equiv_hosts: continue # We found an ESX host, so change it into a name that # monasca recognizes esx_cluster_id = esx_equiv_hosts[hostname] compute_hosts[hostname] = compute_hosts[esx_cluster_id] del compute_hosts[esx_cluster_id] # The last measurement is the latest measurement value_idx = meas['columns'].index('value') compute_hosts[hostname][ui_equiv] = \ meas['measurements'][-1][value_idx] for hostname, meas in compute_hosts.iteritems(): # There are no measurements for a deactivated host, so move on if meas['state'] == 'deactivated': continue # If a host has the USABLE_MEMORY_MB metric, calculate # USED_MEMORY_MB as this is what the UI uses. Normally, the metric # is there unless the host is deactivate or monasca has problems. if USABLE_MEMORY_MB in meas: meas[USED_MEMORY_MB] = \ meas[TOTAL_MEMORY_MB] - meas[USABLE_MEMORY_MB] del compute_hosts[hostname][USABLE_MEMORY_MB] used_mem = float(meas[USED_MEMORY_MB]) total_mem = float(meas[TOTAL_MEMORY_MB]) meas['used_memory_perc'] = 100 * used_mem / total_mem if USED_STORAGE_MB in meas: used_stor = float(meas[USED_STORAGE_MB]) total_stor = float(meas[TOTAL_STORAGE_MB]) meas['used_storage_perc'] = 100 * used_stor / total_stor # Create a mapping to translate nova service_host to hostname host_hyp_dict = {host_info['service_host']: name for name, host_info in compute_hosts.items()} results = {} for clust, host_list in clusters.iteritems(): results[clust] = {} for host in host_list: if host in host_hyp_dict: alt_host = host_hyp_dict[host] # Replace name, id and type from the information in # compute_list2 for key in ('name', 'id', 'type'): compute_hosts[alt_host][key] = \ comp2_dict[host][key] results[clust][host] = compute_hosts[alt_host] return results @classmethod def needs_services(cls): return ['compute', 'monitoring', 'ardana'] 070701000DFF8E000081A40000000000000000000000015B64B4DF00000D37000000FD0000000200000000000000000000005200000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/bll/plugins/ironic_service.py# (c) Copyright 2016-2017 Hewlett Packard Enterprise Development LP # (c) Copyright 2017-2018 SUSE LLC from bll import api from bll.plugins import service from ironicclient import client as ironic_client from bll.plugins.region_client import RegionClient from bll.common.util import get_conf class IronicSvc(service.SvcBase): """ The ``target`` value for this plugin is ``ironic``. See :ref:`rest-api` for a full description of the request and response formats. """ def __init__(self, *args, **kwargs): super(IronicSvc, self).__init__(*args, **kwargs) self.clients = RegionClient('baremetal', self._get_ironic_client, self.token, self.region) def _get_ironic_client(self, region=None, url=None, **kwargs): return ironic_client.get_client( 1, os_auth_token=self.token, ironic_url=url, os_region_name=region, user_agent=api.USER_AGENT, insecure=get_conf("insecure")) @service.expose('node.list') def list_nodes(self): """ Request format:: "target": "ironic", "operation": "node.list" :return: List of nodes from the ironic service """ nodelist = [] for client, region in self.clients.get_clients(): for node in client.node.list(): nodelist.append(node.to_dict()) return nodelist @service.expose('node.get') def get_node(self): """ Get details for a specific node. Request format:: "target": "ironic", "operation": "node.get", "node_id": "MYNODEID" """ for client, region in self.clients.get_clients(): node = client.node.get(node_id=self.data['node_id']) if node: return node.to_dict() @service.expose('baremetal-list') def baremetal_list(self): """ Return a list of nodes from nova and ironic. Details are return from both nova and ironic only for baremetal instances. Request format:: "target": "ironic", "operation": "baremetal-list" """ inst_list = self.call_service(target='nova', operation='instance-list', data={'show_baremetal': True}, region=self.region )['instances'] inst_dict = {item['id']: item for item in inst_list} agg_list = [] for node in self.list_nodes(): details = { 'baremetal': node, 'compute': inst_dict.get(node['instance_uuid']) } agg_list.append(details) return agg_list @service.expose('node.delete') def delete_node(self): """ Deletes a node from ironic. Request format:: "target": "ironic", "operation": "node.delete", "node_id": "MYNODEID" :returns: ``None`` when the operation completes successfully """ for client, region in self.clients.get_clients(): client.node.delete(node_id=self.data['node_id']) @classmethod def needs_services(cls): return ['baremetal'] 070701000DFF8A000081A40000000000000000000000015B64B4DF00003F59000000FD0000000200000000000000000000005300000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/bll/plugins/monitor_service.py# (c) Copyright 2015-2017 Hewlett Packard Enterprise Development LP # (c) Copyright 2017-2018 SUSE LLC import logging from datetime import datetime, timedelta from monascaclient import client as msclient from collections import defaultdict from bll import api from bll.common.util import get_conf from bll.plugins.service import SvcBase, expose from bll.common.exception import InvalidBllRequestException TYPE_UNKNOWN = 'unknown' TYPE_UP = 'up' TYPE_DOWN = 'down' LOG = logging.getLogger(__name__) api_version = '2_0' class MonitorSvc(SvcBase): """ Obtain monitoring information from monasca. The ``target`` value for this plugin is ``monitor``. See :ref:`rest-api` for a full description of the request and response formats. """ # The monasca alarm definition update only permits these keys to be # supplied UPDATEABLE_KEYS = ( 'alarm_id', 'undetermined_actions', 'name', 'alarm_actions', 'match_by', 'expression', 'ok_actions', 'description', 'severity', 'actions_enabled') def __init__(self, *args, **kwargs): """ Set default values for service. """ super(MonitorSvc, self).__init__(*args, **kwargs) self.client = self._get_monasca_client() def _get_monasca_client(self): """ Build the monasca client """ monasca_url = self.token_helper.get_service_endpoint('monitoring') keystone_url = self.token_helper.get_service_endpoint('identity') + 'v3' # All monasca data is stored in the admin project, so get a token # to that project token = self.token_helper.get_token_for_project('admin') return msclient.Client(api_version=api_version, endpoint=monasca_url, token=token, auth_url=keystone_url, project_name='admin', project_domain_name='Default', insecure=get_conf("insecure"), user_agent=api.USER_AGENT) def _get_alarm_data(self): """ Helper function to extract the fields relevant for alarms and replace 'id' with 'alarm_id' :return: """ return self._replace_id(self.request.get_data(), 'alarm_id') def _get_notif_data(self): """ Helper function to extract the fields relevant for notifications and replace 'id' with 'notification_id' :return: """ return self._replace_id(self.request.get_data(), 'notification_id') @expose() def alarm_definition_create(self): return self.client.alarm_definitions.create(**self._get_alarm_data()) @expose() def alarm_definition_delete(self): self.client.alarm_definitions.delete(**self._get_alarm_data()) @expose() def alarm_definition_list(self): return self.client.alarm_definitions.list(**self._get_alarm_data()) @expose() def alarm_definition_patch(self): return self.client.alarm_definitions.patch(**self._get_alarm_data()) @expose() def alarm_definition_show(self): return self.client.alarm_definitions.get(**self._get_alarm_data()) @expose() def alarm_definition_update(self): # Monasca complains if the several of the values returned from # the show are fed back into the update data = self._get_alarm_data() updated_values = {k: v for k, v in data.iteritems() if k in self.UPDATEABLE_KEYS} return self.client.alarm_definitions.update(**updated_values) @expose() def alarm_delete(self): self.client.alarms.delete(**self._get_alarm_data()) @expose() def alarm_history(self): return self.client.alarms.history(**self._get_alarm_data()) @expose() def alarm_history_list(self): return self.client.alarms.history_list(**self._get_alarm_data()) @expose() def alarm_list(self): return self.client.alarms.list(**self._get_alarm_data()) @expose() def alarm_patch(self): return self.client.alarms.patch(**self._get_alarm_data()) @expose() def alarm_show(self): return self.client.alarms.get(**self._get_alarm_data()) @expose() def alarm_update(self): data = self._get_alarm_data() # Monasca complains if some fields from the show # are fed back into the update for key in ('links', 'alarm_definition', 'metrics', 'created_timestamp', 'updated_timestamp', 'state_updated_timestamp'): if key in data: data.pop(key) return self.client.alarms.update(**data) @expose() def measurement_list(self): return self.client.metrics.list_measurements(**self.request.get_data()) @expose() def metric_create(self): return self.client.metrics.create(**self.request.get_data()) @expose() def metric_list(self): return self.client.metrics.list(**self.request.get_data()) @expose() def metric_names(self): return self.client.metrics.list_names(**self.request.get_data()) @expose() def metric_statistics(self): return self.client.metrics.list_statistics(**self.request.get_data()) @expose() def notification_create(self): return self.client.notifications.create(**self._get_notif_data()) @expose() def notification_delete(self): self.client.notifications.delete(**self._get_notif_data()) @expose() def notification_list(self): return self.client.notifications.list(**self._get_notif_data()) @expose() def notification_show(self): return self.client.notifications.get(**self._get_notif_data()) @expose() def notification_update(self): data = self._get_notif_data() # Monasca complains if the links returned from the show # are fed back into the update data.pop('links', None) return self.client.notifications.update(**data) @expose() def notificationtype_list(self): return self.client.notificationtypes.list(**self.request.get_data()) @expose() def get_all_instances_status(self): results = {} for name in ['vm.host_alive_status', 'vm.ping_status']: parms = {} parms['name'] = name parms['start_time'] = \ (datetime.utcnow() - timedelta(minutes=5)).isoformat() parms['group_by'] = '*' parms['dimensions'] = {'component': 'vm'} meas_groups = self.client.metrics.list_measurements(**parms) for meas_info in meas_groups: key = meas_info['dimensions']['resource_id'] if key not in results: results[key] = {} try: meas = meas_info['measurements'] (time, code, value) = meas[-1] results[key][name] = int(code) # Special-case handling for value_meta_data # from host_alive_status if name == 'vm.host_alive_status': results[key]['host_alive_status_value_meta_detail'] = \ value.get('detail', '') except (IndexError, TypeError, KeyError): # Something really bad happened if there are # no measurements results[key][name] = -1 return results @expose() def get_instance_metrics(self): data = self.request.get_data() results = {} if 'instances' not in data or 'metrics' not in data: # you give me nothing, I give you nothing return results # NOTE: We get the last 5 minutes of data and calc an average # for the request metrics. This avoids issues where we # may not receive measurement data for some reason within # the last second or minute or xxx. for instance in data['instances']: instdata = {} for metric in data['metrics']: instdata[metric] = self._get_5min_avg_metric(instance, metric) results[instance] = instdata return results def _get_5min_avg_metric(self, instance, metric): parms = {} parms["dimensions"] = {"resource_id": instance} parms["name"] = metric parms["merge_metrics"] = True try: # get 5 minutes of measurement data parms["start_time"] = \ (datetime.utcnow() - timedelta(minutes=5)).isoformat() measurements = self.client.metrics.list_measurements(**parms) values = measurements[0]['measurements'] # and calculate the mean for this data return sum(secs for secs in (i[1] for i in values)) / len(values) except: # If we couldn't get any data or div by zero or whatever # mark this data as unknown since we couldn't get it # The UI will see this value as 'null' return None def _replace_id(self, data, field): """ The monasca API does not follow the REST standard: the data returned from GETs always has an id field named 'id', but the data expected in PUT, DELETE, and PATCH expects the id field to be named something else: sometimes alarm_id or notification_id. This function can be used to adjust the incoming data to match what the API function expects """ if 'id' not in data: return data result = data.copy() id = result.pop('id') if field not in result: result[field] = id return result def _get_host_measurements(self, observer, hostname, status_data): if not status_data: return [] for status in status_data: host = status['dimensions']['hostname'] obs = status['dimensions']['observer_host'] if host == hostname and obs == observer: return status return [] @expose('get_appliances_status') def get_appliances_status(self): data = self.request.get_data() results = {} if 'hostnames' not in data: return results # get ALL the ping measurements for all hosts start_time = (datetime.utcnow() - timedelta(minutes=5)) \ .strftime("%Y-%m-%dT%H:%M:%SZ") meas_parms = { 'name': 'host_alive_status', "start_time": start_time, 'group_by': "*", 'dimensions': { 'test_type': 'ping' } } ping_measurements = self.client.metrics.list_measurements(**meas_parms) target_hosts = defaultdict(set) all_observers = set() for meas in ping_measurements: hostname = meas['dimensions']['hostname'] observer = meas['dimensions']['observer_host'] target_hosts[hostname].add(observer) all_observers.add(observer) # for each target host, see if it's pingable by any of the observers for hostname in data['hostnames']: # requesting status for a non-existent target host if hostname not in target_hosts: if hostname in all_observers: # This host is an observer(ping source), therefore, if it # can ping other target hosts, then we presume it is up for meas in ping_measurements: observer = meas['dimensions']['observer_host'] if hostname == observer: if self._get_ping_status(meas) == TYPE_UP: # found at least one target host with a good # ping so stop looking and mark the observer # host as good results[hostname] = TYPE_UP break # if we couldn't find a good ping, if not results.get(hostname): results[hostname] = TYPE_UNKNOWN else: # This host is not a ping source or target results[hostname] = TYPE_UNKNOWN else: # requesting status for an existing target host final_ping_status = TYPE_UNKNOWN for observer in target_hosts[hostname]: meas = self._get_host_measurements( observer, hostname, ping_measurements) ping_status = self._get_ping_status(meas) # if it's up, we stop checking and say it's up. if ping_status == TYPE_UP: final_ping_status = TYPE_UP break # if it's down, we say it's down, but keep looking. if ping_status == TYPE_DOWN: final_ping_status = TYPE_DOWN # We ignore TYPE_UNKNOWNs unless they're all TYPE_UNKNOWNs. results[hostname] = final_ping_status return results def _get_ping_status(self, host_meas): if not host_meas or not host_meas['measurements']: return TYPE_UNKNOWN (time, ping_value, value_meta) = host_meas['measurements'][-1] if ping_value == 0.0: return TYPE_UP elif ping_value == 1.0: return TYPE_DOWN else: return TYPE_UNKNOWN @expose('alarm_count') def _get_alarm_count(self): data = self.request.get_data() # Was there a dimension_name_filter given? dim_name_filter_str = data.pop('dimension_name_filter', None) if dim_name_filter_str: dim_name_filters = [str.strip() for str in dim_name_filter_str.split(',')] # make sure dimension dimension_name is in group_by if 'group_by' not in data.keys() \ or 'dimension_name' not in data['group_by']: raise InvalidBllRequestException(self._( 'cannot filter on dimension_name_filter: {} ' 'without a group_by on dimension_name').format( dim_name_filter_str)) else: dim_name_filters = None # Monasca has a 10K limit from alarm-count and we want all of them, so # page thru all of them limit = 10000 offset = 0 all_counts = [] while True: data['limit'] = limit data['offset'] = offset alarm_counts = self.client.alarms.count(**data) columns = alarm_counts['columns'] counts = alarm_counts['counts'] # Handle the situation where we hit a page boundary (i.e. have no # more items at all # This happens when the next page returns something like # counts = [[0, None, None, .....] where the first entry is # 0 and subsequent elements are all 'None' if len(counts) == 1 and \ len(counts[0]) > 1 and \ counts[0][1] is None: break all_counts.extend(counts) if len(counts) < limit: # we got it all, so no need to get any more break offset += limit # Do we need to do any filtering? if dim_name_filters: dim_idx = columns.index('dimension_name') filtered_counts = [c for c in all_counts if c[dim_idx] in dim_name_filters] all_counts = filtered_counts return { 'columns': columns, 'counts': all_counts } @classmethod def needs_services(cls): return ['monitoring'] 070701000DFF8C000081A40000000000000000000000015B64B4DF00004CED000000FD0000000200000000000000000000005000000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/bll/plugins/nova_service.py# (c) Copyright 2015-2017 Hewlett Packard Enterprise Development LP # (c) Copyright 2017-2018 SUSE LLC from collections import Counter from datetime import timedelta, datetime from time import sleep from bll import api from bll.common.util import get_conf from bll.plugins import service from bll.plugins.region_client import RegionClient from novaclient.exceptions import NotFound import logging import requests from novaclient import client as novaclient LOG = logging.getLogger(__name__) # Power states as defined in nova/nova/compute/power_state.py power_states = { 0: 'NO STATE', 1: 'RUNNING', 3: 'PAUSED', 4: 'SHUTDOWN', 6: 'CRASHED', 7: 'SUSPENDED' } # instance list filters FILTER_DBAAS = 'dbaas' FILTER_MSGAAS = 'msgaas' FILTER_CI = 'ci' FILTER_PROJECT = 'project' class NovaSvc(service.SvcBase): """ The ``target`` value for this plugin is ``nova``. See :ref:`rest-api` for a full description of the request and response formats. """ SUGGESTED_POLL_INTERVAL = 5 def __init__(self, *args, **kwargs): """ Initializer for the Nova Client Service """ super(NovaSvc, self).__init__(*args, **kwargs) self.clients = RegionClient('compute', self._get_nova_client, self.token, self.region) def _get_nova_client(self, region=None, **kwargs): try: return novaclient.Client( "2", session=self.token_helper.get_session(), endpoint_type=get_conf("services.endpoint_type", default="internalURL"), region_name=region, user_agent=api.USER_AGENT) except Exception as e: LOG.error("Error creating nova client : %s ", e) raise requests.exceptions.HTTPError(e.message) @service.expose('hypervisor-stats') def hypervisor_stats(self): """ Get the statistics for cpu, memory, and storage across all hypervisors. In a multi-region environment, if no specific region is requested, then data from all regions will be returned. Request format:: "target": "nova", "operation": "hypervisor-stats" """ req_keys = ('vcpus_used', 'vcpus', 'local_gb_used', 'local_gb', 'memory_mb_used', 'memory_mb') sums = {key: 0 for key in req_keys} for client, region in self.clients.get_clients(): stats = client.hypervisors.statistics() for key in req_keys: sums[key] += getattr(stats, key, 0) # TODO: float is probably not necessary, check UI code return { 'used': { 'cpu': sums['vcpus_used'], 'memory': sums['memory_mb_used'], 'storage': sums['local_gb_used'] }, 'total': { 'cpu': float(sums['vcpus']), 'memory': float(sums['memory_mb']), 'storage': sums['local_gb'] } } @service.expose('service-list') def service_list(self): """ Return a list summarizing how many compute nodes are present, how many are up, and how many are in an error state. In a multi-region environment, if no specific region is requested, then data from all regions will be returned. Request format:: "target": "nova", "operation": "service-list" """ compute_list = [] for client, region in self.clients.get_clients(): compute_list.extend(client.services.list(binary="nova-compute")) up_nodes = len( [compute for compute in compute_list if compute.state == "up"]) down_nodes = len(compute_list) - up_nodes return {"ok": up_nodes, "error": down_nodes, "total": len(compute_list)} def _get_historical_data(self, start_date, end_date, resources): startdate = datetime.strptime(start_date, "%Y-%m-%d") enddate = datetime.strptime(end_date, "%Y-%m-%d") resources_count = {} today = 0 while startdate <= enddate: resources_count[enddate.isoformat()] = (today, 0) enddate = enddate + timedelta(days=-1) today = today - 1 for resource in resources: event_date = datetime.strptime(resource[0], "%Y-%m-%d").isoformat() day, count = resources_count[event_date] resources_count[event_date] = day, resource[1] response = sorted( resources_count.values(), key=lambda tup: tup[0]) average = round( (sum(x[1] for x in response) + 0.0) / len(resources_count), 2) data = {'data': response, 'average': average} return data def _deleted_servers(self, start_date): options = {'all_tenants': True, 'deleted': True, "changes-since": start_date, 'sort_key': 'deleted_at', 'sort_dir': 'asc'} deleted = [] for client, region in self.clients.get_clients(): servers = client.servers.list(search_opts=options) for server in servers: output = vars(server) deleted_at = output.get('OS-SRV-USG:terminated_at') or \ output.get('updated') if deleted_at: deleted_at = deleted_at.split("T")[0] deleted.append(deleted_at) deleted = Counter(deleted).items() return deleted def _created_servers(self, start_date): options = {'all_tenants': True, "changes-since": start_date, 'sort_key': 'created_at', 'sort_dir': 'asc'} created = [] for client, region in self.clients.get_clients(): servers = client.servers.list(search_opts=options) for server in servers: output = vars(server) launched_at = output.get('created') or \ output.get('OS-SRV-USG:launched_at') if launched_at: launched_at = launched_at.split("T")[0] if launched_at >= start_date: created.append(launched_at) created = Counter(created).items() return created @service.expose('servers-list') def servers_list(self): """ Get historical information about virtual machines created and deleted in a given time range. In a multi-region environment, if no specific region is requested, then data from all regions will be returned. Request format:: "target": "nova", "operation": "servers-list", "start_date": "2016-01-01T00:00:00", "end_date": "2016-02-01T00:00:00" """ start_date = self.request[api.DATA]['start_date'] end_date = self.request[api.DATA]['end_date'] start_date = datetime.strptime( start_date, "%Y-%m-%d").isoformat().split("T")[0] created_vms = self._created_servers(start_date) deleted_vms = self._deleted_servers(start_date) created_data = self._get_historical_data( start_date, end_date, created_vms) deleted_data = self._get_historical_data( start_date, end_date, deleted_vms) return {'created': created_data, 'deleted': deleted_data} @service.expose('hypervisor-list') def _hypervisor_list(self): """ Get list of hypervisors with details, optionally including their ping status from monasca. In a multi-region environment, if no specific region is requested, then data from all regions will be returned. Request format:: "target": "nova", "operation": "hypervisor-list", "include_status": True """ ret_hyp_list = [] include_status = self.data.get('include_status', True) for client, region in self.clients.get_clients(): hypervisor_list = client.hypervisors.list(detailed=True) for hypervisor in hypervisor_list: hypervisor_data = {} hypervisor_data["allocated_cpu"] = hypervisor.vcpus_used hypervisor_data["total_cpu"] = hypervisor.vcpus hypervisor_data["allocated_memory"] = hypervisor.memory_mb_used hypervisor_data["total_memory"] = hypervisor.memory_mb hypervisor_data["allocated_storage"] = hypervisor.local_gb_used hypervisor_data["total_storage"] = hypervisor.local_gb hypervisor_data["instances"] = hypervisor.running_vms name = getattr(hypervisor, hypervisor.NAME_ATTR, hypervisor.host_ip) hypervisor_data["name"] = name hypervisor_data["hypervisor_id"] = hypervisor.id hypervisor_data["status"] = hypervisor.status hypervisor_data["state"] = hypervisor.state hypervisor_data["type"] = hypervisor.hypervisor_type hypervisor_data["service_host"] = hypervisor.service['host'] hypervisor_data["hypervisor_hostname"] = \ hypervisor.hypervisor_hostname hypervisor_data["region"] = region ret_hyp_list.append(hypervisor_data) # Get host_alive_status:ping results for all compute hosts found if include_status: hostnames = [hd['service_host'] for hd in ret_hyp_list] statuses = self.call_service(target='monitor', operation='get_appliances_status', data={ 'hostnames': hostnames }) # Fill in the ping status details for each of the hosts for hyp in ret_hyp_list: hyp['ping_status'] = \ statuses.get(hyp['service_host'], 'unknown') return ret_hyp_list @service.expose('instance-list') def instance_list(self): """ Get list of instances and their details. If ``show_baremetal`` is set to ``False`` (the default), then baremetal instances will be excluded from the results. ``filter`` can be provided to return only those instances that match the filter, which can have the values: ``dbaas``, ``msgaas``, ``ci``, and ``project``; if ``filter`` is ``project``, then an additional ``project_id`` field should be provided to control that filter. In a multi-region environment, if no specific region is requested, then data from all regions will be returned. Request format:: "target": "nova", "operation": "instance-list", "show_baremetal": True or False, "filter": "dbaas" """ # by default, don't show baremetal instances show_baremetal = self.data.get('show_baremetal', False) proj_list = self.call_service(target='user_group', operation='project_list') proj_dict = {item['id']: item['name'] for item in proj_list} # if the baremetal service is available and we don't want to see # baremetal instances, we'll need a list of baremetal instance uuids # so that they can be filtered from the server list bm_uuid_list = [] baremetal_svc = self.token_helper.get_service_endpoint('baremetal') if baremetal_svc and not show_baremetal: bm_list = self.call_service(target='ironic', operation='node.list', region=self.region) bm_uuid_list = [bmi['instance_uuid'] for bmi in bm_list] # Create a list for the UI to use instance_list = [] search_opts = {'all_tenants': True} for client, region in self.clients.get_clients(): server_list = client.servers.list(search_opts=search_opts, limit=-1, detailed=True) # flavors tell us cpu/memory/disk allocated to the instance flavor_list = client.flavors.list(is_public=None, detailed=True) flavor_dict = {flavitem.id: flavitem for flavitem in flavor_list} image_list = [] if hasattr(client, 'images'): image_list = client.images.list() elif hasattr(client, 'glance'): image_list = client.glance.list() image_dict = {imgitem.id: imgitem for imgitem in image_list} for nova_inst in server_list: # filter out any baremetal instances if nova_inst.id in bm_uuid_list: continue instance = {} instance['name'] = nova_inst.name instance['status'] = nova_inst.status instance['host'] = nova_inst._info['OS-EXT-SRV-ATTR:host'] instance['availability_zone'] = \ nova_inst._info['OS-EXT-AZ:availability_zone'] instance['id'] = nova_inst.id try: instance['image'] = image_dict[nova_inst.image['id']].name except: # There are some instances tied to non-existent images instance['image'] = 'UNKNOWN' instance['addresses'] = nova_inst.addresses instance['created'] = nova_inst.created powernum = nova_inst._info['OS-EXT-STS:power_state'] instance['power_state'] = power_states.get( powernum, "UNKNOWN[%d]" % powernum) # tasks states defined in nova/nova/compute/task_states.py instance['task_state'] = \ nova_inst._info['OS-EXT-STS:task_state'] instance['key_name'] = nova_inst.key_name instance['metadata'] = nova_inst.metadata # get the project name. If it's not found, just get the # tenant id instead instance['project'] = proj_dict.get(nova_inst.tenant_id, nova_inst.tenant_id) instance['tenant_id'] = nova_inst.tenant_id instance['region'] = region try: flavor = flavor_dict[nova_inst.flavor['id']] instance['flavor'] = flavor.name instance['cpu'] = {'vcpus': flavor.vcpus} instance['memory'] = {'ram': flavor.ram} instance['storage'] = {'disk': flavor.disk} except: # There are some instances tied to flavors that don't # appear to show up in flavor-list instance['flavor'] = None instance['cpu'] = None instance['memory'] = None instance['storage'] = None self._populate_metrics(instance) instance_list.append(instance) return {'instances': instance_list} def _populate_metrics(self, instance): monasca_metrics = self.request[api.DATA].get('monasca_metrics') if isinstance(monasca_metrics, list): instance['metrics'] = dict() for metric in monasca_metrics: req_data = \ dict(self.request[api.DATA].get('monasca_data')) req_data['name'] = metric monasca_dimensions = \ self.request[api.DATA].get('monasca_dimensions') if isinstance(monasca_dimensions, dict): req_data['dimensions'] = dict() for key, value in monasca_dimensions.iteritems(): # if dict its more complicated if isinstance(value, dict): instanceKey = value.get('property') req_data['dimensions'][key] = \ instance[instanceKey] else: req_data['dimensions'][key] = value res = self.call_service( target='monitor', operation=req_data.get('operation', 'metric_statistics'), data=req_data ) instance['metrics'][metric] = res @service.expose('service-delete') def service_delete(self): """ Delete the service which can be specified either by ``novaid`` or ``hostname``. Request format:: "target": "nova", "operation": "service-delete", "novaid": "ID" """ nova_id = self.data.get('novaid') host_name = self.data.get('hostname') if not nova_id and not host_name: raise Exception("Either novaid or hostname must be " "populated") found = False for client, region in self.clients.get_clients(): compute_list = client.services.list(binary="nova-compute") for instance in compute_list: if (nova_id and instance.id == nova_id) or \ (not nova_id and host_name == instance.host): client.services.delete(instance.id) # Also break out of the outer loop when the host is found found = True break if found: break if not found: raise Exception(self._("Could not find service to delete")) return self._( "Executed nova service delete on service {}").format(nova_id) @service.expose('instance-delete', is_long=True) def server_delete(self, validate): """ Delete an instance. This function will not return until nova has completed the deletion of the instance, which is to say, when nova no longer returns it in its list of servers. Request format:: "target": "nova", "operation": "instance-delete", "instance_id": "ID" """ inst_id = self.data['instance_id'] if validate: for client, region in self.clients.get_clients(): # TODO: Test that this doesn't throw an exception if client.servers.get(inst_id): client.servers.delete(inst_id) self.client_used_for_delete = client break self.update_job_status('in progress', 25, task_id=inst_id) return self.SUGGESTED_POLL_INTERVAL else: # Now we call 'nova show' on that instance until it no longer # exists or something bad happens (keystone token times out, # nova has a problem, etc) while True: try: self.client_used_for_delete.servers.get(inst_id) sleep(self.SUGGESTED_POLL_INTERVAL) except NotFound: # No longer see the instance, so we're good! break self.response[api.DATA] = self._("instance {} deleted").format( inst_id) return self.response @classmethod def needs_services(cls): return ['compute'] 070701000DFF88000081A40000000000000000000000015B64B4DF0000AC66000000FD0000000200000000000000000000006100000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/bll/plugins/objectstorage_summary_service.py# (c) Copyright 2015-2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017-2018 SUSE LLC try: from builtins import range except ImportError: from __builtin__ import range import logging import copy from datetime import datetime, timedelta from monascaclient import client from bll import api from bll.common.util import get_conf from bll.plugins.service import SvcBase, expose LOG = logging.getLogger(__name__) api_version = '2_0' class ObjectStorageSummarySvc(SvcBase): """ Retrieve summaries of Object Storage monitoring data. The ``target`` value for this plugin is ``objectstorage_summary``. See :ref:`rest-api` for a full description of the request and response formats. The services in this file are all specific queries to monasca and have peculiar response formats that are tightly coupled with the UI screens that call them. All time values should be specified in UTC and should follow the ISO-8601 date/time format, for example: 2016-12-25T00:00:00Z . """ def __init__(self, *args, **kwargs): super(ObjectStorageSummarySvc, self).__init__(*args, **kwargs) self.monasca_client = self._get_monasca_client() def _get_monasca_client(self): """ Build the monasca client """ monasca_url = self.token_helper.get_service_endpoint('monitoring') keystone_url = self.token_helper.get_service_endpoint('identity') + 'v3' # All monasca data is stored in the admin project, so get a token # to that project token = self.token_helper.get_token_for_project('admin') return client.Client(api_version=api_version, endpoint=monasca_url, token=token, auth_url=keystone_url, project_name='admin', project_domain_name='Default', insecure=get_conf("insecure"), user_agent=api.USER_AGENT) def _get_time_series(self, end_time, interval, ret_fields_mapping, period): resp_dict = {} ret_list = [] te = datetime.strptime(end_time, "%Y-%m-%dT%H:%M:%SZ") ts = te + timedelta(hours=-int(interval)) for fields in ret_fields_mapping.iteritems(): local_fields = copy.deepcopy(fields[1]) local_fields["start_time"] = ts.strftime("%Y-%m-%dT%H:%M:%SZ") local_fields["period"] = int(period) local_fields["end_time"] = end_time resp_dict = self.monasca_client.metrics.list_statistics( **local_fields) ret_list.append(resp_dict) return ret_list def _get_monasca_aggregated_data(self, end_time, interval, ret_fields_mapping): resp_dict = {} ret_list = [] te = datetime.strptime(end_time, "%Y-%m-%dT%H:%M:%SZ") ts = te + timedelta(hours=-int(interval)) start_time = ts.strftime("%Y-%m-%dT%H:%M:%SZ") for fields in ret_fields_mapping.iteritems(): local_fields = copy.deepcopy(fields[1]) local_fields["start_time"] = start_time local_fields["end_time"] = end_time resp_dict = self.monasca_client.metrics.list_measurements( **local_fields) ret_list.append(resp_dict) return ret_list def _get_monasca_formatted_data(self, output, end_time, interval, period): spec = {} interval = int(interval) for i in output: try: stat = i[0]['statistics'] if interval == 1: spec[i[0]['name']] = stat[-1][1] else: spec[i[0]['name']] = self._handle_monasca_no_values( end_time, stat, interval, period) except (TypeError, IndexError, KeyError): pass return spec def _get_file_system_mount_point(self, cluster, hostname): ret_list = [] metric_list = \ self.monasca_client.metrics.list( name="swiftlm.systems.check_mounts", dimensions={"service": "object-storage", "hostname": hostname, "cluster": cluster}) for metric in metric_list: ret_list.append(metric["dimensions"]["mount"]) return ret_list def total_node(self): clusterwise_data = self.call_service(target="catalog", operation="get_swift_clusters") resp_dict = {} for i in clusterwise_data.iteritems(): resp_dict[i[0].split(":")[1]] = i[1] return resp_dict @expose() def storage(self): end_time = self.request[api.DATA][api.DATA]["end_time"] interval = self.request[api.DATA][api.DATA]["interval"] cluster = self.request[api.DATA][api.DATA]["cluster"] hostname = self.request[api.DATA][api.DATA]["hostname"] period = self.request[api.DATA][api.DATA]["period"] ret_fields_mapping = { "percentage_usage": { "name": "swiftlm.diskusage.host.val.usage", "dimensions": {"service": "object-storage", "cluster": cluster, "hostname": hostname }, "statistics": "max", "merge_metrics": "true" }, "used": { "name": "swiftlm.diskusage.host.val.used", "dimensions": {"service": "object-storage", "cluster": cluster, "hostname": hostname }, "statistics": "max", "merge_metrics": "true" }, "total_usage": { "name": "swiftlm.diskusage.host.val.size", "dimensions": {"service": "object-storage", "cluster": cluster, "hostname": hostname }, "statistics": "max", "merge_metrics": "true" } } output = self._get_time_series(end_time, interval, ret_fields_mapping, period) return self._get_monasca_formatted_data(output, end_time, interval, period) @expose() def memory(self): end_time = self.request[api.DATA][api.DATA]["end_time"] interval = self.request[api.DATA][api.DATA]["interval"] cluster = self.request[api.DATA][api.DATA]["cluster"] hostname = self.request[api.DATA][api.DATA]["hostname"] period = self.request[api.DATA][api.DATA]["period"] ret_fields_mapping = { "total": { "name": "mem.total_mb", "dimensions": {"service": "system", "cluster": cluster, "hostname": hostname }, "statistics": "max", "merge_metrics": "true" }, "free": { "name": "mem.free_mb", "dimensions": {"service": "system", "cluster": cluster, "hostname": hostname }, "statistics": "max", "merge_metrics": "true" }, } output = self._get_time_series(end_time, interval, ret_fields_mapping, period) return self._get_monasca_formatted_data( output, end_time, interval, period) @expose() def load_average_donut(self): end_time = self.request[api.DATA][api.DATA]["end_time"] interval = self.request[api.DATA][api.DATA]["interval"] cluster = self.request[api.DATA][api.DATA]["cluster"] hostname = self.request[api.DATA][api.DATA]["hostname"] period = self.request[api.DATA][api.DATA]["period"] ret_fields_mapping = { "load_five_avg": { "name": "swiftlm.load.host.val.five", "dimensions": {"service": "object-storage", "cluster": cluster, "hostname": hostname }, "statistics": "max", "merge_metrics": "true" } } output = self._get_time_series(end_time, interval, ret_fields_mapping, period) return self._get_monasca_formatted_data( output, end_time, interval, period) @expose() def time_to_replicate(self): end_time = self.request[api.DATA][api.DATA]["end_time"] interval = self.request[api.DATA][api.DATA]["interval"] period = self.request[api.DATA][api.DATA]["period"] ret_fields_mapping = { "replication_object_duration": { "name": "swiftlm.replication.cp.avg.object_duration", "dimensions": {"service": "object-storage"}, "statistics": "max", "merge_metrics": "true" }, "replication_account_duration": { "name": "swiftlm.replication.cp.avg.account_duration", "dimensions": {"service": "object-storage"}, "statistics": "max", "merge_metrics": "true" }, "replication_container_duration": { "name": "swiftlm.replication.cp.avg.container_duration", "dimensions": {"service": "object-storage"}, "statistics": "max", "merge_metrics": "true" } } output = self._get_time_series(end_time, interval, ret_fields_mapping, period) return self._get_monasca_formatted_data( output, end_time, interval, period) @expose() def oldest_replication_completion(self): end_time = self.request[api.DATA][api.DATA]["end_time"] interval = self.request[api.DATA][api.DATA]["interval"] period = self.request[api.DATA][api.DATA]["period"] ret_fields_mapping = { "replication_object_last": { "name": "swiftlm.replication.cp.max.object_last", "dimensions": {"service": "object-storage"}, "statistics": "max", "merge_metrics": "true" }, "replication_account_last": { "name": "swiftlm.replication.cp.max.account_last", "dimensions": {"service": "object-storage"}, "statistics": "max", "merge_metrics": "true" }, "replication_container_last": { "name": "swiftlm.replication.cp.max.container_last", "dimensions": {"service": "object-storage"}, "statistics": "max", "merge_metrics": "true" } } output = self._get_time_series(end_time, interval, ret_fields_mapping, period) return self._get_monasca_formatted_data( output, end_time, interval, period) @expose() def current_capacity(self): end_time = self.request[api.DATA][api.DATA]["end_time"] interval = self.request[api.DATA][api.DATA]["interval"] period = self.request[api.DATA][api.DATA]["period"] ret_fields_mapping = { "total_size": {"name": "swiftlm.diskusage.val.size_agg", "dimensions": {"host": "all"}, "statistics": "max", "merge_metrics": "true"}, "avail_size": {"name": "swiftlm.diskusage.val.avail_agg", "dimensions": {"host": "all"}, "statistics": "max", "merge_metrics": "true"}} output = self._get_time_series(end_time, interval, ret_fields_mapping, period) return self._get_monasca_formatted_data( output, end_time, interval, period) @expose() def rate_of_change(self): end_time = self.request[api.DATA][api.DATA]["end_time"] interval = self.request[api.DATA][api.DATA]["interval"] period = self.request[api.DATA][api.DATA]["period"] ret_fields_mapping = { "diskusage-used": { "name": "swiftlm.diskusage.rate_agg", "dimensions": {"host": "all"}, "statistics": "max", "merge_metrics": "true" } } output = self._get_time_series(end_time, interval, ret_fields_mapping, period) try: stat = output[0][0]['statistics'] if int(interval) == 1 or int(interval) == 2: if len(stat) > 1: return [stat[-1][1] - stat[-2][1]] else: return [stat[-1][1]] else: value = [] for j in range(0, len(stat) - 1): temp = [] temp.append(stat[j][0]) temp.append(int(stat[j + 1][1]) - int(stat[j][1])) value.append(temp) return (self._handle_monasca_no_values(end_time, value, interval, period)) except (TypeError, IndexError, KeyError): return [-1] @expose() def filesystem_utilization(self): end_time = self.request[api.DATA][api.DATA]["end_time"] interval = self.request[api.DATA][api.DATA]["interval"] cluster = self.request[api.DATA][api.DATA]["cluster"] hostname = self.request[api.DATA][api.DATA]["hostname"] period = self.request[api.DATA][api.DATA]["period"] ret_fields_mapping = { "filesystem_utilization_min": { "name": "swiftlm.diskusage.host.min.usage", "dimensions": {"service": "object-storage", "hostname": hostname, "cluster": cluster }, "merge_metrics": "true", "statistics": "min" }, "filesystem_utilization_max": { "name": "swiftlm.diskusage.host.max.usage", "dimensions": {"service": "object-storage", "hostname": hostname, "cluster": cluster }, "merge_metrics": "true", "statistics": "max" }, "filesystem_utilization_main": { "name": "swiftlm.diskusage.host.val.usage", "dimensions": {"service": "object-storage", "cluster": cluster, "hostname": hostname }, "statistics": "max", "merge_metrics": "true" } } output = self._get_time_series(end_time, interval, ret_fields_mapping, period) return self._get_monasca_formatted_data( output, end_time, interval, period) @expose() def latency_healthcheck(self): end_time = self.request[api.DATA][api.DATA]["end_time"] interval = self.request[api.DATA][api.DATA]["interval"] period = self.request[api.DATA][api.DATA]["period"] ret_fields_mapping = { "healthcheck_latency_avg": { "name": "swiftlm.umon.target.avg.latency_sec", "dimensions": { "component": "healthcheck-api", "service": "object-storage" }, "statistics": "avg", "merge_metrics": "true" }, "healthcheck_latency_min": { "name": "swiftlm.umon.target.min.latency_sec", "dimensions": { "component": "healthcheck-api", "service": "object-storage" }, "statistics": "min", "merge_metrics": "true" }, "healthcheck_latency_max": { "name": "swiftlm.umon.target.max.latency_sec", "dimensions": { "component": "healthcheck-api", "service": "object-storage" }, "statistics": "max", "merge_metrics": "true" } } output = self._get_time_series(end_time, interval, ret_fields_mapping, period) return self._get_monasca_formatted_data( output, end_time, interval, period) @expose() def latency_operational(self): end_time = self.request[api.DATA][api.DATA]["end_time"] interval = self.request[api.DATA][api.DATA]["interval"] period = self.request[api.DATA][api.DATA]["period"] ret_fields_mapping = { "operational_latency_avg": { "name": "swiftlm.umon.target.avg.latency_sec", "dimensions": { "component": "rest-api", "service": "object-storage"}, "statistics": "avg", "merge_metrics": "true" }, "operational_latency_min": { "name": "swiftlm.umon.target.min.latency_sec", "dimensions": { "component": "rest-api", "service": "object-storage"}, "statistics": "min", "merge_metrics": "true" }, "operational_latency_max": { "name": "swiftlm.umon.target.max.latency_sec", "dimensions": { "component": "rest-api", "service": "object-storage"}, "statistics": "max", "merge_metrics": "true" } } output = self._get_time_series(end_time, interval, ret_fields_mapping, period) return self._get_monasca_formatted_data( output, end_time, interval, period) @expose() def async_pending(self): end_time = self.request[api.DATA][api.DATA]["end_time"] interval = self.request[api.DATA][api.DATA]["interval"] period = self.request[api.DATA][api.DATA]["period"] ret_fields_mapping = { "async_pending_max": { "name": "swiftlm.async_pending.cp.total.queue_length", "dimensions": { "service": "object-storage" }, "statistics": "max", "merge_metrics": "true" } } output = self._get_time_series(end_time, interval, ret_fields_mapping, period) return self._get_monasca_formatted_data( output, end_time, interval, period) @expose() def alarms(self): resp_dict = self.monasca_client.alarms.count( metric_dimensions={"service": "object-storage"}, group_by="state") return resp_dict @expose() def alarm_description(self): cluster = self.request[api.DATA][api.DATA]["cluster"] hostname = self.request[api.DATA][api.DATA]["hostname"] ids = [] resp_dict = {} final_dict = {} alarms_list = \ self.monasca_client.alarms.list( metric_dimensions={"service": "object-storage", "hostname": hostname, "cluster": cluster}) for alarm in alarms_list: ids.append(alarm['id']) for id in ids: id = str(id) try: resp_dict[id] = self.monasca_client.alarms.get(alarm_id=id) final_dict[id] = {} final_dict[id]['name'] = \ resp_dict[id]['alarm_definition']['name'] final_dict[id]['severity'] = \ resp_dict[id]['alarm_definition']['severity'] final_dict[id]['alarm_definition_id'] = \ resp_dict[id]['alarm_definition']['id'] final_dict[id]['state'] = \ resp_dict[id]['state'] resp_dict[id]['details'] = \ self.monasca_client.alarm_definitions.get( alarm_id=final_dict[id]['alarm_definition_id']) final_dict[id]['description'] = \ resp_dict[id]['details']['description'] if resp_dict[id]['state'] == "UNDETERMINED": final_dict[id]['status'] = "UNKNOWN" elif resp_dict[id]['state'] == "OK": final_dict[id]['status'] = "OK" elif resp_dict[id]['state'] == "ALARM" and \ resp_dict[id]['alarm_definition'][ 'severity'] in ("CRITICAL", "HIGH"): final_dict[id]['status'] = "CRITICAL" elif resp_dict[id]['state'] == "ALARM" and \ resp_dict[id]['alarm_definition'][ 'severity'] in ("MEDIUM", "LOW"): final_dict[id]['status'] = "WARNING" except (TypeError, IndexError, KeyError): final_dict[id] = {} return final_dict @expose() def file_systems(self): end_time = self.request[api.DATA][api.DATA]["end_time"] interval = self.request[api.DATA][api.DATA]["interval"] cluster = self.request[api.DATA][api.DATA]["cluster"] hostname = self.request[api.DATA][api.DATA]["hostname"] period = self.request[api.DATA][api.DATA]["period"] resp_dict = {} mount_point = self._get_file_system_mount_point(cluster, hostname) for mount in mount_point: ret_fields_mapping = { "total": { "name": "swiftlm.diskusage.host.val.size", "dimensions": {"service": "object-storage", "hostname": hostname, "cluster": cluster, "mount": mount }, "statistics": "max", "merge_metrics": "true" }, "used": { "name": "swiftlm.diskusage.host.val.used", "dimensions": {"service": "object-storage", "hostname": hostname, "cluster": cluster, "mount": mount }, "statistics": "max", "merge_metrics": "true" }, "percentage_utilized": { "name": "swiftlm.diskusage.host.val.usage", "dimensions": {"service": "object-storage", "hostname": hostname, "cluster": cluster, "mount": mount }, "statistics": "max", "merge_metrics": "true" }, "mount_status": { "name": "swiftlm.systems.check_mounts", "dimensions": {"service": "object-storage", "cluster": cluster, "hostname": hostname, "mount": mount }, "statistics": "max", "merge_metrics": "true" } } output = self._get_time_series(end_time, interval, ret_fields_mapping, period) resp_dict[mount] = self._get_monasca_formatted_data(output, end_time, interval, period) return resp_dict @expose() def mount_status(self): end_time = self.request[api.DATA][api.DATA]["end_time"] interval = self.request[api.DATA][api.DATA]["interval"] cluster = self.request[api.DATA][api.DATA]["cluster"] hostname = self.request[api.DATA][api.DATA]["hostname"] period = self.request[api.DATA][api.DATA]["period"] resp_dict = {} resp_dict['mount_status'] = dict(mounted=0, unmounted=0) mount_point = self._get_file_system_mount_point(cluster, hostname) mount_point_flag = False resp_dict['total_mount_point'] = len(mount_point) for mount in mount_point: ret_fields_mapping = { "mount_status": { "name": "swiftlm.systems.check_mounts", "dimensions": {"service": "object-storage", "cluster": cluster, "hostname": hostname, "mount": mount }, "statistics": "max", "merge_metrics": "true" } } output = self._get_time_series(end_time, interval, ret_fields_mapping, period) for i in output: try: stat = i[0]['statistics'] if stat[-1][1] == 0.0: resp_dict['mount_status']['mounted'] = \ resp_dict['mount_status']['mounted'] + 1 elif stat[-1][1] == 2.0: resp_dict['mount_status']['unmounted'] = \ resp_dict['mount_status']['unmounted'] + 1 except (TypeError, IndexError, KeyError): continue mount_point_flag = True if not mount_point_flag: return dict(mounted=-1, unmounted=-1, total_mount_point=-1) return resp_dict @expose() def service_availability(self): """ Request format:: "target": "objectstorage_summary", "operation": "service_availability", """ end_time = self.request[api.DATA][api.DATA]["end_time"] interval = self.request[api.DATA][api.DATA]["interval"] period = self.request[api.DATA][api.DATA]["period"] ret_fields_mapping = { "service_availability": { "name": "swiftlm.umon.target.val.avail_day", "dimensions": {"component": "rest-api", "service": "object-storage"}, "statistics": "max", "merge_metrics": "true" } } output = self._get_time_series(end_time, interval, ret_fields_mapping, period) if output: return self._get_monasca_formatted_data( output, end_time, interval, period) else: return {} @expose() def load_average(self): end_time = self.request[api.DATA][api.DATA]["end_time"] interval = self.request[api.DATA][api.DATA]["interval"] period = self.request[api.DATA][api.DATA]["period"] ret_fields_mapping = { "load_avg": { "name": "swiftlm.load.cp.avg.five", "dimensions": {"service": "object-storage"}, "statistics": "avg", "merge_metrics": "true" }, "load_min": { "name": "swiftlm.load.cp.min.five", "dimensions": {"service": "object-storage"}, "statistics": "min", "merge_metrics": "true" }, "load_max": { "name": "swiftlm.load.cp.max.five", "dimensions": {"service": "object-storage"}, "statistics": "max", "merge_metrics": "true" } } output = self._get_time_series(end_time, interval, ret_fields_mapping, period) return self._get_monasca_formatted_data( output, end_time, interval, period) @expose(is_long=True) def heat_map_utilization_focused_inventory(self): end_time = self.request[api.DATA][api.DATA]["end_time"] interval = self.request[api.DATA][api.DATA]["interval"] cluster_node_info = self.total_node() resp_dict = {} metric_name_size = "swiftlm.diskusage.val.size_agg" metric_name_avail = "swiftlm.diskusage.val.avail_agg" for cluster, host_list in cluster_node_info.iteritems(): resp_dict[cluster] = {} for host in host_list: resp_dict[cluster][host] = {} ret_fields_mapping = { "diskusage_used": {"name": metric_name_size, "dimensions": {"aggregation_period": "hourly", "host": host}}, "diskusage_avail": {"name": metric_name_avail, "dimensions": {"aggregation_period": "hourly", "host": host}} } output = self._get_monasca_aggregated_data(end_time, interval, ret_fields_mapping) try: for mon_data in output: resp_dict[cluster][host][mon_data[0]["name"]] = ( mon_data[0]["measurements"][-1][1]) except (TypeError, IndexError, KeyError): resp_dict[cluster][host] = -1 self.update_job_status(percentage_complete=60) return resp_dict @expose(is_long=True) def heat_map_cpu_load_average(self): end_time = self.request[api.DATA][api.DATA]["end_time"] interval = self.request[api.DATA][api.DATA]["interval"] period = self.request[api.DATA][api.DATA]["period"] cluster_node_info = self.total_node() final = {} for cluster, host_list in cluster_node_info.iteritems(): final[cluster] = {} for host in host_list: ret_fields_mapping = { "load_five_avg": { "name": "swiftlm.load.host.val.five", "dimensions": {"service": "object-storage", "cluster": cluster, "hostname": host }, "statistics": "max", "merge_metrics": "true" }} output = self._get_time_series(end_time, interval, ret_fields_mapping, period) try: host_value = self._get_monasca_formatted_data( output, end_time, interval, period) if not host_value: final[cluster][host] = -1 else: final[cluster][host] = \ host_value['swiftlm.load.host.val.five'] except (TypeError, IndexError, KeyError): final[cluster][host] = -1 self.update_job_status(percentage_complete=60) return final @expose(is_long=True) def node_state(self): cluster_node_info = self.total_node() final_dict = {} total_nodes = {"red": 0, "yellow": 0, "green": 0, "grey": 0, "nodes": 0} for cluster, host_list in cluster_node_info.iteritems(): final_dict[cluster] = {"red": 0, "green": 0, "grey": 0, "yellow": 0, "nodes": 0} for host in host_list: resp_dict = self.monasca_client.alarms.count( metric_dimensions={"service": "object-storage", "hostname": host, "cluster": cluster}, group_by="state,severity") try: total_nodes["nodes"] += 1 severity_dict = {"red": 0, "green": 0, "grey": 0, "yellow": 0} alarms = resp_dict['counts'] for value, alarm, severity in alarms: if alarm == "UNDETERMINED": severity_dict["grey"] += 1 elif alarm == "ALARM" and severity in ("HIGH", "CRITICAL"): severity_dict["red"] += 1 elif alarm == "ALARM" and severity in ("MEDIUM", "LOW"): severity_dict["yellow"] += 1 elif alarm == "OK": severity_dict["green"] += 1 else: severity_dict["grey"] += 1 except (TypeError, IndexError, KeyError): severity_dict["grey"] += 1 for severity in ('red', 'yellow', 'green', 'grey'): if severity_dict[severity] > 0: final_dict[cluster][severity] += 1 final_dict[cluster]["nodes"] += 1 total_nodes[severity] += 1 break final_dict["total_nodes"] = total_nodes self.update_job_status(percentage_complete=60) return final_dict @expose(is_long=True) def health_focused(self): cluster_node_info = self.total_node() final_dict = {} for cluster, host_list in cluster_node_info.iteritems(): final_dict[cluster] = {} for host in host_list: resp_dict = self.monasca_client.alarms.count( metric_dimensions={"service": "object-storage", "hostname": host, "cluster": cluster}, group_by="state,severity") try: final_dict[cluster][host] = {"red": 0, "green": 0, "grey": 0, "yellow": 0} alarms = resp_dict['counts'] for value, alarm, severity in alarms: if alarm == "UNDETERMINED": final_dict[cluster][host]["grey"] += int(value) elif alarm == "ALARM" and severity in ("HIGH", "CRITICAL"): final_dict[cluster][host]["red"] += int(value) elif alarm == "ALARM" and severity in ("MEDIUM", "LOW"): final_dict[cluster][host]["yellow"] += ( int(value)) elif alarm == "OK": final_dict[cluster][host]["green"] += int(value) except (TypeError, IndexError, KeyError): final_dict[cluster][host]["grey"] = -1 self.update_job_status(percentage_complete=60) return final_dict def _get_keystone_projects(self, ids=None): # ids= ["monasca", "admin"] OR "None" # return [{"name":"admin", "id":"5678h6"}, # {"name":"monasca", "id":"4673h2"}] ks_projects = self.call_service(target="user_group", operation="project_list") if not ids or ids == "None": return ks_projects else: return [p for p in ks_projects if p['name'] in ids] @expose() def project_list(self): """ Returns a list of projects from keystone. .. deprecated:: 1.0 Use :py:meth:`~.UserGroupSvc.get_project_list` instead. The ``ids`` field in the request can either be a list of projects of the string ``"None"`` (which will return all projects) Request format:: "target": "objectstorage_summary", "operation": "project_list", "ids": "None" Response format:: "data":[ {"name": "admin", "id": "8e3a82a61ff74d84ab52aafcc6249e71"}, {"name": "demo", "id": "19239852352358e9a89f989fa7a9aaee"} ... ] """ try: ids = self.request[api.DATA][api.DATA]["ids"] return self._get_keystone_projects(ids) except Exception as e: LOG.exception("Error occurred: %s" % e) def _get_project_capacity(self, project_id, end_time, interval, period): # Project capacity for given project metric_name = None if project_id == "all": metric_name = "storage.objects.size_agg" else: metric_name = "storage.objects.size" ret_fields_mapping = { "replication_object_duration": { "name": metric_name, "dimensions": {"project_id": project_id}, "statistics": "max", "merge_metrics": "true" } } return self._get_time_series(end_time, interval, ret_fields_mapping, period) @expose(is_long=True) def topten_project_capacity(self): """ Returns the top ten projects sorted by storage capacity. Request format:: "target": "objectstorage_summary", "operation": "topten_project_capacity", "end_time": "2016-12-25T00:00:00Z", "interval": "72", "period": "3600" Response format:: "data": [ {"swift-monitor": {"id": "857dedf742e94deaad5591720649898c", "value": -1}}, {"monitor": {"id": "91aa65fb59d9416aac6d62d20a7e8064", "value": -1}}, ... ] """ try: end_time = self.request[api.DATA][api.DATA]["end_time"] interval = self.request[api.DATA][api.DATA]["interval"] period = self.request[api.DATA][api.DATA]["period"] proj_capacities = [] for project in self._get_keystone_projects(): capacity = self._get_project_capacity(project['id'], end_time, interval, period) try: val = capacity[0][0]['statistics'][-1][-1] except (TypeError, IndexError, KeyError): val = -1 proj_cap = {project['name']: {"id": project['id'], "value": val}} proj_capacities.append(proj_cap) # Sort by value field in each project's dictionary sorted_projects = sorted(proj_capacities, key=lambda c: c.values()[0]['value'], reverse=True) self.update_job_status(percentage_complete=60) return sorted_projects[:10] except Exception as e: LOG.exception("Error occurred: %s" % e) def _handle_monasca_no_values(self, end_time, monasca_statistics, interval, period): """ Given a [timestamp, measurement_value] from monasca, this method assigns default -1 value for all the timestamps within the range of start and end time where monasca does not give any value """ format_out = "%Y-%m-%dT%H:%M:%SZ" format_in = "%Y-%m-%dT%H:%M:%S.%fZ" # end_time should be formatted without subsecond precision end = datetime.strptime(end_time, format_out) curr = end - timedelta(hours=float(interval)) increment = timedelta(seconds=float(period)) statistics = [] stat = iter(monasca_statistics) next_avail = stat.next() next_avail_end = datetime.strptime(next_avail[0], format_in) while curr <= end: if curr < next_avail_end: statistics.append([curr.strftime(format_out), -1]) else: statistics.append(next_avail) try: next_avail = stat.next() next_avail_end = datetime.strptime(next_avail[0], format_in) except StopIteration: # set next_avail_end beyond the end time of interest next_avail_end = end + increment curr += increment return statistics @expose(is_long=True) def project_capacity(self): id = self.request[api.DATA][api.DATA]["id"] end_time = self.request[api.DATA][api.DATA]["end_time"] interval = self.request[api.DATA][api.DATA]["interval"] period = self.request[api.DATA][api.DATA]["period"] capacity = self._get_project_capacity(id, end_time, interval, period) self.update_job_status(percentage_complete=60) try: stats = capacity[0][0]['statistics'] if int(interval) == 1: return [int(stats[0][-1])] else: return self._handle_monasca_no_values( end_time, stats, interval, period) except (TypeError, IndexError, KeyError): return [-1] @expose(is_long=True) def project_capacity_roc(self): id = self.request[api.DATA][api.DATA]["id"] end_time = self.request[api.DATA][api.DATA]["end_time"] interval = self.request[api.DATA][api.DATA]["interval"] period = self.request[api.DATA][api.DATA]["period"] capacity = self._get_project_capacity(id, end_time, interval, period) self.update_job_status(percentage_complete=60) try: stats = capacity[0][0]['statistics'] if int(interval) == 2: return [int(stats[-1][1]) - int(stats[-2][1])] else: value = [] for i in range(0, len(stats) - 1): temp = [] temp.append(stats[i][0]) temp.append(int(stats[i + 1][1]) - int(stats[i][1])) value.append(temp) return self._handle_monasca_no_values(end_time, value, interval, period) except (TypeError, IndexError, KeyError): return [-1] @classmethod def needs_services(cls): # Even though this module does not use swift directly, it should # be suppressed if swift is not running, since the monitoring data # would be void and meaningless return ['monitoring', 'swift'] 070701000DFF8D000081A40000000000000000000000015B64B4DF00000C81000000FD0000000200000000000000000000005700000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/bll/plugins/preferences_service.py# (c) Copyright 2015-2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC import json from bll.plugins import service import logging import pecan import pymysql.cursors LOG = logging.getLogger(__name__) class PreferencesSvc(service.SvcBase): """ Simple service to manage user preferences. User preferences are stored as JSON in a mysql database. The ``target`` value for this plugin is ``preferences``. See :ref:`rest-api` for a full description of the request and response formats. """ def __init__(self, *args, **kwargs): super(PreferencesSvc, self).__init__(*args, **kwargs) config = pecan.conf.db.to_dict() config['cursorclass'] = pymysql.cursors.DictCursor self.connection = pymysql.connect(**config) @service.expose(action='GET') def _get(self): return self._get_mysql(self.data.get("user")) @service.expose(action='POST') def _post(self): self._post_mysql(self.data.get("user"), self.data.get("prefs")) @service.expose(action='PUT') def _put(self): self._put_mysql(self.data.get("user"), self.data.get("prefs")) @service.expose(action='DELETE') def _delete(self): self._delete_mysql(self.data.get("user")) # Functions for writing def _get_mysql(self, user): with self.connection.cursor() as cursor: sql = "SELECT `prefs` from `preferences` WHERE `username`=%s" cursor.execute(sql, user) row = cursor.fetchone() cursor.close() if row is None: message = self._("User {} does not exist").format(user) LOG.warn(message) self.response.error(message) return prefs = row.get("prefs") if isinstance(prefs, dict): return prefs return json.loads(prefs) def _post_mysql(self, user, prefs): with self.connection.cursor() as cursor: sql = "INSERT INTO `preferences` (`username`, `prefs`) " + \ "VALUES (%s,%s)" cursor.execute(sql, [user, json.dumps(prefs)]) cursor.close() self.connection.commit() def _put_mysql(self, user, prefs): with self.connection.cursor() as cursor: sql = "select count(*) from preferences where username=%s" cursor.execute(sql, user) user_found = (cursor.fetchone()['count(*)'] == 1) if user_found: sql = "UPDATE `preferences` SET `prefs`=%s WHERE `username`=%s" cursor.execute(sql, [json.dumps(prefs), user]) cursor.close() self.connection.commit() if not user_found: message = self._( "Cannot update non-existent user {}").format(user) LOG.warn(message) self.response.error(message) def _delete_mysql(self, user): with self.connection.cursor() as cursor: sql = "DELETE FROM `preferences` WHERE `username`=%s" cursor.execute(sql, user) cursor.close() self.connection.commit() 070701000DFF8F000081A40000000000000000000000015B64B4DF00000818000000FD0000000200000000000000000000005100000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/bll/plugins/region_client.py# (c) Copyright 2015-2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC from bll.api.auth_token import TokenHelpers class RegionClient(object): """ Creates and returns openstack clients for the region(s) specified in the request. Since client creation may be expensive, possibly requiring a round trip to keystone, care is taken to avoid creating clients until necessary and to avoid creating multiple clients for the same URL """ def __init__(self, endpoint_type, create_func, token, region): self.endpoint_type = endpoint_type self.create_func = create_func self.token = token self.region = region self.clients = [] def _get_endpoints(self): helper = TokenHelpers(self.token) endpoints = helper.get_endpoints(self.endpoint_type, self.region) client_list = [] for e in endpoints: client_list.append({ 'endpoint': e, 'client': None }) return client_list def get_client(self): """ Returns a single client. This is useful for services like keystone or monasca where all regions share a single instance. """ clients = self.get_clients() # return just the first client from the generator return clients.next() def get_clients(self): """ Generator that returns clients and the region name, one per invocation, depending on the region specified in the request. If no region is specified (region is ``None``), then all regions will be returned. Otherwise, the generator will return a client for the specified region. """ if not self.clients: self.clients = self._get_endpoints() for c in self.clients: region = c['endpoint']['region'] url = c['endpoint']['url'] if not c['client']: c['client'] = self.create_func(region=region, url=url) yield (c['client'], region) 070701000DFF90000081A40000000000000000000000015B64B4DF00005535000000FD0000000200000000000000000000004B00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/bll/plugins/service.py# (c) Copyright 2015-2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC import logging import inspect import pykka import copy import time import traceback from bll.common import util from bll.common.exception import InvalidBllRequestException, BllException from bll.api.auth_token import TokenHelpers from bll.api.response import BllResponse from bll.api.request import BllRequest from bll import api from bll.common.job_status import get_job_status, update_job_status from bll.common import i18n from bll.common.util import context, new_txn_id from stevedore import driver from requests.exceptions import HTTPError LOG = logging.getLogger(__name__) def expose(operation=None, action='GET', is_long=False): """ A decorator for exposing methods as BLL operations/actions Keyword arguments: * operation the name of the operation that the caller should supply in the BLL request. If this optional argument is not supplied, then the operation name will be the name of the method being decorated. * action the name of the action (typically ``GET``, ``PUT``, ``DELETE`` or ``POST``) that will be matched against the BLL request. If this optional argument is not supplied, then the dispatching mechanism will ignore the action. * is_long indicates that the method being decorated is a long-running function. Long-running methods may either be called once (via complete) or twice (via handle for validation and via for the rest) depending on their signature -- if the method has an argument (other than self), then it will be called twice, in which case the parameter will be populated with a boolean indicating whether it is being called via handle; in this case, the function should return a recommended polling interval. Note, if you override handle or complete, this decorations will be ignored! When a normal (short-running) method is called, its return value should just be the data portion of the response. The handle method will take care of building a full BllResponse structure and placing this return value in the data portion. Otherwise, if there are any problems, the method should just throw an appropriate exception. The handle method will catch the exception and package it up in an appropriate BllResponse object. The decorator can be applied multiple times to the same function, in order to permit a function to be called via multiple operations. For example:: class MySvc(SvcBase) @expose('add_stuff') @expose('update_stuff') def add_or_update(self): ... """ def decorate(f): f.exposed = True if not hasattr(f, 'operation'): f.operation = [] f.operation.append(operation or f.__name__) if action: f.action = action if is_long: f.is_long = is_long # normally a decorator returns a wrapped function, but here # we return f unmodified, after registering it return f return decorate class SvcBase(pykka.ThreadingActor): """ Base class for plugins. """ def __init__(self, bll_request=None): super(SvcBase, self).__init__() # Extract common fields into attributes if not isinstance(bll_request, BllRequest): raise InvalidBllRequestException('Invalid request class') self.request = bll_request self.response = BllResponse(self.request) if api.AUTH_TOKEN in self.request: self.token_helper = TokenHelpers(self.request.get(api.AUTH_TOKEN)) self.token = self.request.get(api.AUTH_TOKEN) request_data = self.request.get(api.DATA, {}) self.action = self.request.get(api.ACTION) self.api_version = None self.operation = None self.data = {} self.txn_id = self.request.txn_id self.region = self.request.get(api.REGION) # Assign _ as a member variable in this plugin class for localizing # messages self._ = i18n.get_(self.request.get(api.LANGUAGE, 'en')) # Extract request data, filtering out known keys for k, v in request_data.iteritems(): if k == api.OPERATION: self.operation = v elif k == 'suggest_sync': # Omit the obsolete, poorly-supported suggest_sync flag pass elif k == api.VERSION: self.api_version = v else: self.data[k] = v @staticmethod def spawn_service(bll_request): """ Call the targeted service, using stevedore to load the plugin whose name matches the 'target' field of the incoming request. """ srv = None # Assign _ in this function for localizing messages _ = i18n.get_(bll_request.get(api.LANGUAGE, 'en')) try: mgr = driver.DriverManager( namespace='bll.plugins', name=bll_request.get(api.TARGET)) srv = mgr.driver.start(bll_request=bll_request).proxy() reply = srv.sc_handle() result = reply.get() srv.sc_complete() return result except Exception as e: LOG.exception('spawn_service failed') if srv is not None: srv.stop() response = BllResponse(bll_request) if isinstance(e, HTTPError): message = e.message elif isinstance(e, BllException): # Localize the overview field from the BllException prefix = _(e.overview) message = _("{0}: {1}").format(prefix, e) else: message = "%s" % e response.error(message.rstrip()) return response def handle(self): """ Handle the request by dispatching the request to the appropriate method. Override this method if desired, to implement your own dispatching and execution of short-running work. """ if not self.operation and not self.action: raise InvalidBllRequestException(self._( "Operation and action missing")) method = self._get_method(self.operation, self.action) if method is None: raise InvalidBllRequestException( self._("Unsupported operation: {}").format(self.operation)) if getattr(method, 'is_long', False): self.response[api.PROGRESS] = dict(percentComplete=0) self.response[api.STATUS] = api.STATUS_INPROGRESS polling_interval = 10 if method.im_func.func_code.co_argcount > 1: # If the long-running method expects an argument, call it # set to True and expect a polling interval in return polling_interval = method(True) or polling_interval self.response[api.POLLING_INTERVAL] = \ getattr(self, api.POLLING_INTERVAL, 10) self.update_job_status(percentage_complete=0) return self.response data = method() # In cases where we don't have the data in the response, it # had better be in the return value. # i.e. compute_summary_service: resource_history() if not self.response[api.DATA]: self.response[api.DATA] = data self.response[api.PROGRESS] = dict(percentComplete=100) self.response.complete() return self.response def complete(self): """ Complete the request. Override this method and do long running processing here. """ if not self.operation and not self.action: return method = self._get_method(self.operation, self.action) if method is None: return if getattr(method, 'is_long', False): try: if method.im_func.func_code.co_argcount > 1: # If the long-running method expects an argument, call it # set to False to indicate that it is being called # during complete response = method(False) else: response = method() # Permit the calling function to just return a normal # value, and then just add it to the 'data' element of the # existing self.response if isinstance(response, BllResponse): self.response = response else: self.response[api.DATA] = response self.response[api.PROGRESS] = dict(percentComplete=100) self.response.complete() except Exception as e: self.response.error("%s" % e) return self.response def _get_method(self, operation=None, action=None): """ Use inspection to get the name of the @exposed function that corresponds to the operation and action being requested If there is only one method whose exposed name matches the operation, then that method is returned, regardless of the action. If there is more, then the action will be consulted to decide which to return. """ candidates = [] # Find all candidates -- those members whose name matches the # operation, ignoring the action. for name, f in inspect.getmembers(self, inspect.ismethod): # Only look at those that are exposed if not getattr(f, 'exposed', False): continue # If operation is specified, the function must expose that op op_list = getattr(f, 'operation', []) if operation and operation not in op_list: continue candidates.append((name, f)) if not candidates: return # In most cases, there is only a single function with the given # operation, so we will return that. if len(candidates) == 1: name, f = candidates[0] return f # If action is specified, the function must expose that action for name, f in candidates: if action == getattr(f, 'action', None): return f def sc_handle(self): """ Handle the request. Called by the SvcCollection class. Do not override, this method. Override method handle. """ context.txn_id = self.request.txn_id reply = self.handle() return copy.deepcopy(reply) def sc_complete(self): """ complete the request. Called by the SvcCollection class. Do not override this method. Override method 'complete'. """ try: bll_response = self.complete() if bll_response is not None: self.put_resource(self.request.txn_id, bll_response) except Exception as e: LOG.exception('sc_complete failed.') self.response.exception(traceback.format_exc()) self.put_resource(self.request.txn_id, self.response.error( "%s" % e)) finally: self.stop() def update_job_status(self, msg=None, percentage_complete=0, txn_id=None, **kwargs): if percentage_complete is not None: self.response[api.PROGRESS] = {api.PERCENT_COMPLETE: percentage_complete} if msg: self.response[api.DATA] = msg self.response.update(**kwargs) txn = txn_id or self.txn_id update_job_status(txn, self.response) def put_resource(self, txn_id, msg): update_job_status(txn_id, msg) @classmethod def is_available(cls, available_services): """ Returns a boolean to indicate whether this plugin is available, i.e., that all of the dependent services and requirements that this plugin needs are available. The function is supplied a lists of openstack services from keystone that are available. This check will not be called each time the plugin is executed, so it can afford to be somewhat slow. It is expected to only be called when the client (UI) requests a list of available plugins. """ needs = cls.needs_services() for service in needs: if service not in available_services: return False return True @classmethod def needs_services(cls): """ List of services that this plugin uses, if any. When populated by the plugin, the default implementation of is_available (above) can easily check whether all needed services are available by consulting this function. Note that it is generally unnecessary to indicate that keystone is required for a given plugin since keystone already has to be running for the operations console to even let a user login. """ return [] def call_service_async(self, request=None, target=None, auth_token=None, operation=None, action=None, data=None, region=None, polling_interval=None, max_polls=0, offset=0, scale=1.0, **kwargs): """ Call an asynchronous service in another plugin, and wait for it to complete before returning. If a request is supplied, its values will be used as the basis of the call, and any other parameters will override the values in the request. If a request is not supplied, a new BllRequest will be constructed from the other parameters. This function shields the caller from the complexities of the standard reply mechanism (which returns a dictionary with status values and deeply embedded return data), and offers a more pythonic interface where the data is returned directly from the function, and exceptions are thrown in the event of errors. This function should only be called from within an asynchronous service; calling an asynchronous service from a synchronous service would probably cause the calling operation to time out. The asynchronous service being called may return a percent complete value. If so, then update_job_status will be called to update the percentage complete of the calling service. The percentage value of the called function will be scaled and offset added so that callers which perform multiple steps will have their percentages updated correctly. :param request: Request dictionary or BllRequest (optional) :param target: Target service to invoke (optional) :param token: Authorization token to use (optional). If neither token nor request is specified, then the token from the current service will be used. :param operation: Operation in the service to call (optional) :param action: Action in the service to use (optional) :param data: Data object to place into the request (optional) :param region: Restrict operation to the given region (optional) :param polling_interval: Timeout, in seconds, to wait between polls for status from the async request (optional) :param max_polls: Limit of the number of polls to be attempted. If 0 or not specified, then attempts will be unlimited. :param offset: Value added to percentComplete of target function when updating status of calling plugin (optional) :param scale: Value multiplied by percentComplete of target function when updating status of calling plugin (optional) :param \*\*kwargs: Additional items to pass to the BllRequest constructor (optional) :return: The results of the :func:`handle` method of the called service """ if polling_interval is None: try: polling_interval = request[api.POLLING_INTERVAL] except (AttributeError, TypeError, KeyError): polling_interval = 10 request = self._build_request(request, target, auth_token, operation, action, data, region, **kwargs) handle_reply = SvcBase.spawn_service(request) txn_id = handle_reply.get(api.TXN_ID) pct_key = ".".join((api.PROGRESS, api.PERCENT_COMPLETE)) poll = 0 reply = get_job_status(txn_id) while reply.get(api.STATUS) in (api.STATUS_INPROGRESS, api.STATUS_NOT_FOUND): if max_polls > 0 and poll < max_polls: raise Exception(self._("Timed out waiting for {}").format( request)) pct = util.get_val(reply, pct_key) if pct: overall_pct = offset + scale * pct self.update_job_status(percentage_complete=overall_pct) time.sleep(polling_interval) reply = get_job_status(txn_id) poll += 1 data = reply.get(api.DATA) if reply.get(api.STATUS) == api.STATUS_ERROR: # extract the error message and throw it try: message = data[0][api.DATA] except (TypeError, IndexError, KeyError): message = data or self._("Failure calling {} service").format( target) raise Exception(message) pct = util.get_val(reply, pct_key) if pct: overall_pct = offset + scale * pct self.update_job_status(percentage_complete=overall_pct) return data def call_service(self, request=None, target=None, auth_token=None, operation=None, action=None, data=None, region=None, **kwargs): """ Call a synchronous service in another plugin, i.e. just the handle method. If a request is supplied, its values will be used as the basis of the call, and any other parameters will override the values in the request. If a request is not supplied, a new BllRequest will be constructed from the other parameters. :param request: Request dictionary or BllRequest (optional) :param target: Target service to invoke (optional) :param token: Authorization token to use (optional). If neither token nor request is specified, then the token from the current service will be used. :param operation: Operation in the service to call (optional) :param action: Action in the service to use (optional) :param data: Data object to place into the request (optional) :param region: Restrict operation to the given region (optional) :param \*\*kwargs: Additional items to pass to the BllRequest constructor (optional) :return: The results of the :func:`handle` method of the called service """ request = self._build_request(request, target, auth_token, operation, action, data, region, **kwargs) response = SvcBase.spawn_service(request) if response[api.STATUS] == api.COMPLETE: return response[api.DATA] try: message = response[api.DATA][0][api.DATA] except KeyError: message = response.get(api.DATA) raise Exception(message) def _build_request(self, request=None, target=None, auth_token=None, operation=None, action=None, data=None, region=None, **kwargs): """ Construct a request object using the following, in priority order ( highest priority first): 1. Function arguments (e.g. operation) 2. Values in the ``request`` argument 3. Values from the service making the request for fields that are commonly inherited (txn, region, auth_token) For example, if the auth_token is passed as a parameter, it will be used; otherwise, the auth_token will be taken from any request object passes as a parameter; otherwise it will be copied from the calling service. """ txn_id = new_txn_id(self.txn_id) language = self.request.get(api.LANGUAGE) if not region: if request: region = request.get(api.REGION) region = region or self.region req = BllRequest(request=request, target=target, auth_token=auth_token, operation=operation, action=action, data=data, txn_id=txn_id, region=region, language=language, **kwargs) if not req.get(api.AUTH_TOKEN) and getattr(self, 'token_helper', None): req[api.AUTH_TOKEN] = self.token_helper.get_user_token() return req 070701000DFF85000081A40000000000000000000000015B64B4DF00001692000000FD0000000200000000000000000000005600000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/bll/plugins/user_group_service.py# (c) Copyright 2015-2017 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC import copy from bll import api from bll.common.exception import InvalidBllRequestException from bll.common.util import get_conf, get_service_tenant_name from bll.plugins import service from keystoneclient.v3 import client as ksclient # default OpsConsole Role = Admin ADMIN = 'admin' # monasca-user role required to access some items in UI MONASCA = 'monasca-user' class UserGroupSvc(service.SvcBase): """ This service provides for managing users, groups, and projects. The ``target`` value for this plugin is ``user_group``. See :ref:`rest-api` for a full description of the request and response formats. """ def __init__(self, *args, **kwargs): """ Set default values for service. """ super(UserGroupSvc, self).__init__(*args, **kwargs) self.client = self._get_ks_client() # a helper method to make this more unit-testable def _get_ks_client(self): return ksclient.Client(session=self.token_helper.get_domain_session(), endpoint_type=get_conf("services.endpoint_type", default="internalURL"), interface=get_conf("services.interface", default="internal"), user_agent=api.USER_AGENT) @service.expose('identity_backend') def _get_identity_backend(self): return False @service.expose('users_remove') def _del_user_list(self): users = self.data.get('user_ids', []) for user in users: self.client.users.delete(user) @service.expose('user_update') def _update_user_op(self): user_id = self.data.get('user_id') username = self.data.get('username') email = self.data.get('email') password = self.data.get('password') if not user_id: return if username or email or password: data = {} if username: data['name'] = username if email: data['email'] = email if password: data['password'] = password self.client.users.update(user_id, **data) @service.expose('user_add') def _add_users_op(self): user = self.data.get('username') password = self.data.get('password') email = self.data.get('email') project_name = self.data.get('project_name') projects = self.client.projects.list() for project in projects: if project.name == project_name: project_id = project.id break else: raise InvalidBllRequestException(self._( "Invalid project: {}").format(project_name)) keystone_user = self.client.users.create(name=user, password=password, email=email, project_id=project_id) self._add_role_to_user(keystone_user, ADMIN, project_id) # Monasca user required for dashboard stats self._add_role_to_user(keystone_user, MONASCA, project_id) return copy.copy(keystone_user.id.encode('ascii', 'ignore')) # because we are using the client the user and project are keystone # client classes. def _add_role_to_user(self, user, role_name, project): roles = self.client.roles.list() role_to_add = None for role in roles: if role.name == role_name: role_to_add = role break self.client.roles.grant(role_to_add, user=user, project=project) @service.expose('get_default_project') def _get_default_project(self): projects = self.ks.get_project_list() return self._select_default_project(projects) def _select_default_project(self, projects): # Return the appropriate default project for creating users. For # historical reasons, this is generally 'demo', if present. Otherwise # return any project other than admin and the default services project. if 'demo' in projects: return 'demo' candidates = list(projects) if 'admin' in candidates: candidates.remove('admin') service_project = get_service_tenant_name() if service_project in candidates: candidates.remove(service_project) if candidates: return candidates[0] else: return None @service.expose('users_list') def _get_user_list(self): user_list = [] seen_users = set() projects = self.client.projects.list() for project in projects: for user in self.client.users.list(project_id=project.id): if user.id not in seen_users: user_list.append({'username': user.name, 'project_id': project.id, 'project_name': project.name, 'email': getattr(user, 'email', ''), 'user_id': user.id}) seen_users.add(user.id) return user_list @service.expose('project_list') def get_project_list(self): """ Returns the project list. Duh. """ project_list = [] for project in self.client.projects.list(): project_list.append({ 'id': project.id, 'name': project.name, }) return project_list 070701000DFF9F000081A40000000000000000000000015B64B4DF0000022D000000FD0000000200000000000000000000004300000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/bll/schema.sql-- (c) Copyright 2015-2016 Hewlett Packard Enterprise Development LP -- (c) Copyright 2017 SUSE LLC CREATE TABLE IF NOT EXISTS `preferences` ( `username` VARCHAR(255) NOT NULL, `prefs` TEXT NOT NULL, PRIMARY KEY (`username`) ); CREATE TABLE IF NOT EXISTS `jobs` ( `id` VARCHAR(255) NOT NULL, `updated_at` DATETIME NOT NULL, `status` TEXT NOT NULL, PRIMARY KEY (`id`) ); DROP TABLE IF EXISTS `install`; DROP TABLE IF EXISTS `plugin`; DROP TABLE IF EXISTS `eula`; 07070100081037000041ED0000000000000000000000035B64B4DF00000000000000FD0000000200000000000000000000003800000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/doc07070100081038000041ED0000000000000000000000035B64B4DF00000000000000FD0000000200000000000000000000003F00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/doc/source0707010008104F000081A40000000000000000000000015B64B4DF00000A93000000FD0000000200000000000000000000004700000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/doc/source/conf.py# (c) Copyright 2015-2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import os import sys # ROOT of source distro is 3 levels above this file ROOT = os.path.normpath(os.path.join(os.path.abspath(__file__), "..", "..", "..")) sys.path.insert(0, ROOT) # -- General configuration ---------------------------------------------------- # Add any Sphinx extension module names here, as strings. They can be # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom ones. extensions = [ 'sphinx.ext.autodoc', 'sphinx.ext.viewcode', ] # autodoc generation is a bit aggressive and a nuisance when doing heavy # text edit cycles. # execute "export SPHINX_DEBUG=1" in your terminal to disable # The suffix of source filenames. source_suffix = '.rst' # The master toctree document. master_doc = 'index' # General information about the project. project = 'BLL' copyright = '2016 Hewlett Packard Enterprise Development LP, 2017 SUSE LLC' # If true, '()' will be appended to :func: etc. cross-reference text. add_function_parentheses = True # If true, the current module name will be prepended to all description # unit titles (such as .. function::). # add_module_names = True # The name of the Pygments (syntax highlighting) style to use. pygments_style = 'sphinx' # -- Options for HTML output -------------------------------------------------- # The theme to use for HTML and HTML Help pages. Major themes that come with # Sphinx are currently 'default' and 'sphinxdoc'. # html_theme_path = ["."] # html_theme = '_theme' # html_static_path = ['static'] # Output file base name for HTML help builder. htmlhelp_basename = 'BLLdoc' # Grouping the document tree into LaTeX files. List of tuples # (source start file, target name, title, author, documentclass # [howto/manual]). latex_documents = [ ('index', 'BLL.tex', u'Operations Console Business Logic Layer Documentation', u'Hewlett Packard Enterprise', 'manual'), ] # Example configuration for intersphinx: refer to the Python standard library. # intersphinx_mapping = {'http://docs.python.org/': None} 07070100081051000081A40000000000000000000000015B64B4DF0000017C000000FD0000000200000000000000000000004900000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/doc/source/index.rst.. (c) Copyright 2015-2016 Hewlett Packard Enterprise Development LP (c) Copyright 2017 SUSE LLC Operations Console Business Logic Layer documentation ===================================================== Contents: .. toctree:: :maxdepth: 2 :glob: readme pluginapi restapi services Indices and tables ================== * :ref:`genindex` * :ref:`search` 0707010008104E000081A40000000000000000000000015B64B4DF00000163000000FD0000000200000000000000000000004D00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/doc/source/pluginapi.rst.. (c) Copyright 2016 Hewlett Packard Enterprise Development LP (c) Copyright 2017 SUSE LLC Plugin API ========== SvcBase ------- .. automodule:: bll.plugins.service :members: :undoc-members: :show-inheritance: RegionClient ------------ .. automodule:: bll.plugins.region_client :members: :undoc-members: :show-inheritance: 0707010008104D000081A40000000000000000000000015B64B4DF0000007D000000FD0000000200000000000000000000004A00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/doc/source/readme.rst.. (c) Copyright 2016 Hewlett Packard Enterprise Development LP (c) Copyright 2017 SUSE LLC .. include:: ../../README.rst 07070100081053000081A40000000000000000000000015B64B4DF0000225C000000FD0000000200000000000000000000004B00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/doc/source/restapi.rst.. (c) Copyright 2016 Hewlett Packard Enterprise Development LP (c) Copyright 2017 SUSE LLC .. _rest-api: REST API ======== General ------- The BLL is registered as an endpoint in keystone with the name ``opsconsole``, and the URL registered there is the one to use. This is typically ``https://``\ *HOST*\ ``:9095/api/v1`` in a production environment. The typical headers for json REST calls to OpenStack services are required: | ``Accept: application/json`` | ``Content-Type: application/json`` | ``X-Auth-Token:`` *TOKEN* .. _request-format: Request Format -------------- All REST calls to the BLL are http ``POST`` requests. The payload is a simple json structure with the following items: * ``target`` The BLL plugin that will service the request. This field is required for new requests (i.e. those that are not job status requests) * ``operation`` The operation that will be executed within the BLL. * ``action`` Some operations strive to be REST-like and expect one of the standard REST operations in this field (``GET``, ``POST``, ``PUT`` or ``DELETE``). The action is only needed be a small number of operations in the BLL. * ``region`` A region name can be supplied in a multi-region environment for those requests that need a region. This parameter is not needed in a single-region environment. * ``job_status_request`` (boolean) To obtain the status of a long-running request, set the ``job_status_request`` entry to ``true``. When using this option, the transaction id (``txn_id``) is a required parameter. * ``txn_id`` The transaction id to be used in a job status request. A transaction id can also be supplied for new requests, but this usage is deprecated. * other operation-specific data Depending on the operation, additional parameters may be needed in the request. In the past, these additional elements, along with the ``operation``, were required to be placed into a nested ``data`` dictionary, but that usage is deprecated and discouraged. .. _response-format: Response Format --------------- The response from BLL requests is a json structure with the following items: * ``txn_id`` The transaction id generated by the request. For long-running tasks, this value will be used in subsequent requests to obtain the job status. * ``status`` Status of the request: ``inprogress``, ``complete``, or ``fail``. If the status is ``inprogress``, a ``job_status_request`` should be made to obtain the status of the given transaction. * ``progress`` A json structure containing the field ``percent_complete``, which is a number between 0 and 100 indicating how much of the work is done on the request. * ``recommended_polling_interval`` For long-running tasks, this is the recommended number of seconds to wait between job status requests. * ``data`` The data response of the request. In an error scenario, this will contain an array with a single element containing a dictionary with the error message. This unnecessary nesting will be removed in a future modification. Examples -------- There are a number browser plugins available for Chrome and Firefox for executing REST requests directly in the browser and viewing the traffic. IntelliJ and eclipse also have plugins for this as well. .. highlight:: bash The most common command-line tool is ``curl``. An example of obtaining a token and making a request:: # Obtain a token $ curl https://192.168.245.10:9095/api/v1/auth_token \ -H 'Content-Type: application/json' \ -H 'Accept: application/json' \ --data '{"username":"dev","password":"test"}' {"token": "af95fef3f7684dafa861628040a0fd21", "expires": "2016-09-12T23:37:51.892928+00:00"} # Get a project list $ curl https://192.168.245.10:9095/api/v1/bll \ -H 'Content-Type: application/json' \ -H 'Accept: application/json' \ -H 'X-Auth-Token: af95fef3f7684dafa861628040a0fd21' \ --data '{"target":"user_group","operation":"project_list"}' {"status": "complete", "txn_id": "a22e958f-b943-4e58-a221-cee912dfab41","starttime": 1473709738.243234, "duration": 1.4179999828338623, "progress": {"percentComplete": 100}, "endtime": 1473709739.661234, "data": [{"id": "0290918d30b245d0a39eb06a921e2dc3", "name": "admin"}, {"id": "091676b8f9594149be5e38f457dc7d77", "name": "glance-check"}, {"id": "131c45a104594e5ea9710b1c99aba4b7", "name": "kronos"}, {"id": "2cdec4259bee401f9cea3de90d32685d", "name": "services"}, {"id": "3358b3415e34414697fbcf6105704740", "name": "backup"}, {"id": "69943f70f4bd40debc746151bc5ae93f", "name": "swift-monitor"}, {"id": "699905facfa0479692a35336340b603e", "name": "cinderinternal"}, {"id": "719bf578ef9f4083959224b71e918d1f", "name": "swift-dispersion"}, {"id": "cdf1817b257249bb8c25d6c5d41b9b6c", "name": "octavia"}, {"id": "d55e31ca836547b48d28fbf897f78bf1", "name": "monitor"}, {"id": "d7952e7769274b61b8a1615939da6c6f", "name": "glance-swift"}, {"id": "f862ce8915474f12813866e0b845f971", "name": "demo"}]} # Create a new user $ curl https://192.168.245.10:9095/api/v1/bll \ -H 'Content-Type: application/json' \ -H 'Accept: application/json' \ -H 'X-Auth-Token: af95fef3f7684dafa861628040a0fd21' \ --data '{"target":"user_group","operation":"user_add","username":"john","password":"doe","project_name":"admin"}' {"status": "complete", "txn_id": "c9c05b00-c2ae-4575-a1c2-243de3f1c87d", "starttime": 1473710432.326117, "duration": 1.0908639430999756, "progress": {"percentComplete": 100}, "endtime": 1473710433.416981, "data": "008a0058d70b4ff39da8e80ff076f428"} A better command-line tool that has intuitive command arguments and format JSON output for humans is `httpie `_. Here is a session using `http-prompt `_ that obtains the same results:: # Obtain a token $ http-prompt https://192.168.245.10:9095/api/v1/bll Version: 0.5.0 https://192.168.245.10:9095/api/v1/bll> post ../auth_token username=dev password=test { "expires": "2016-09-12T23:37:51.892928+00:00", "token": "af95fef3f7684dafa861628040a0fd21" } https://192.168.245.10:9095/api/v1/bll> X-Auth-Token:af95fef3f7684dafa861628040a0fd21 # Get a project list https://192.168.245.10:9095/api/v1/bll> post target=user_group operation=project_list { "data": [ { "id": "0290918d30b245d0a39eb06a921e2dc3", "name": "admin" }, { "id": "091676b8f9594149be5e38f457dc7d77", "name": "glance-check" }, { "id": "131c45a104594e5ea9710b1c99aba4b7", "name": "kronos" }, { "id": "2cdec4259bee401f9cea3de90d32685d", "name": "services" }, { "id": "3358b3415e34414697fbcf6105704740", "name": "backup" }, { "id": "69943f70f4bd40debc746151bc5ae93f", "name": "swift-monitor" }, { "id": "699905facfa0479692a35336340b603e", "name": "cinderinternal" }, { "id": "719bf578ef9f4083959224b71e918d1f", "name": "swift-dispersion" }, { "id": "cdf1817b257249bb8c25d6c5d41b9b6c", "name": "octavia" }, { "id": "d55e31ca836547b48d28fbf897f78bf1", "name": "monitor" }, { "id": "d7952e7769274b61b8a1615939da6c6f", "name": "glance-swift" }, { "id": "f862ce8915474f12813866e0b845f971", "name": "demo" } ], "duration": 0.4173750877380371, "endtime": 1473710137.649522, "progress": { "percentComplete": 100 }, "starttime": 1473710137.232147, "status": "complete", "txn_id": "39d3ef12-3b55-49c6-baa0-4dad62e8c4bf" } # Create a user https://192.168.245.10:9095/api/v1/bll> post target=user_group operation=user_add username=john password=doe project_name=admin { "data": "1713464877664fc69aad0bd1c0b5a80c", "duration": 1.4430599212646484, "endtime": 1473710330.7217, "progress": { "percentComplete": 100 }, "starttime": 1473710329.27864, "status": "complete", "txn_id": "965c2b27-0981-4c99-abfc-daeabef8a993" } 070701000A47C2000041ED0000000000000000000000025B64B4DF00000000000000FD0000000200000000000000000000004700000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/doc/source/service070701000A47C5000081A40000000000000000000000015B64B4DF000000BC000000FD0000000200000000000000000000005200000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/doc/source/service/ardana.rst.. (c) Copyright 2016 Hewlett Packard Enterprise Development LP (c) Copyright 2017 SUSE LLC Ardana Service -------------- .. autoclass:: bll.plugins.ardana_service.ArdSvc :members: 070701000A47CC000081A40000000000000000000000015B64B4DF000000C3000000FD0000000200000000000000000000005300000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/doc/source/service/catalog.rst.. (c) Copyright 2016 Hewlett Packard Enterprise Development LP (c) Copyright 2017 SUSE LLC Catalog Service --------------- .. autoclass:: bll.plugins.catalog_service.CatalogSvc :members: 070701000A47CA000081A40000000000000000000000015B64B4DF000000BF000000FD0000000200000000000000000000005200000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/doc/source/service/cinder.rst.. (c) Copyright 2016 Hewlett Packard Enterprise Development LP (c) Copyright 2017 SUSE LLC Cinder Service -------------- .. autoclass:: bll.plugins.cinder_service.CinderSvc :members: 070701000A47CD000081A40000000000000000000000015B64B4DF000000C3000000FD0000000200000000000000000000005300000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/doc/source/service/compute.rst.. (c) Copyright 2016 Hewlett Packard Enterprise Development LP (c) Copyright 2017 SUSE LLC Compute Service --------------- .. autoclass:: bll.plugins.compute_service.ComputeSvc :members: 070701000A47C6000081A40000000000000000000000015B64B4DF000000B7000000FD0000000200000000000000000000005000000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/doc/source/service/eula.rst.. (c) Copyright 2016 Hewlett Packard Enterprise Development LP (c) Copyright 2017 SUSE LLC Eula Service ------------ .. autoclass:: bll.plugins.eula_service.EulaSvc :members: 070701000A47C4000081A40000000000000000000000015B64B4DF000000BF000000FD0000000200000000000000000000005200000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/doc/source/service/ironic.rst.. (c) Copyright 2016 Hewlett Packard Enterprise Development LP (c) Copyright 2017 SUSE LLC Ironic Service -------------- .. autoclass:: bll.plugins.ironic_service.IronicSvc :members: 070701000A47C9000081A40000000000000000000000015B64B4DF000000C3000000FD0000000200000000000000000000005300000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/doc/source/service/monitor.rst.. (c) Copyright 2016 Hewlett Packard Enterprise Development LP (c) Copyright 2017 SUSE LLC Monitor Service --------------- .. autoclass:: bll.plugins.monitor_service.MonitorSvc :members: 070701000A47C8000081A40000000000000000000000015B64B4DF000000B7000000FD0000000200000000000000000000005000000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/doc/source/service/nova.rst.. (c) Copyright 2016 Hewlett Packard Enterprise Development LP (c) Copyright 2017 SUSE LLC Nova Service ------------ .. autoclass:: bll.plugins.nova_service.NovaSvc :members: 070701000A47C3000081A40000000000000000000000015B64B4DF000000FC000000FD0000000200000000000000000000006100000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/doc/source/service/objectstorage_summary.rst.. (c) Copyright 2016 Hewlett Packard Enterprise Development LP (c) Copyright 2017 SUSE LLC Object Storage Summary Service ------------------------------ .. autoclass:: bll.plugins.objectstorage_summary_service.ObjectStorageSummarySvc :members: 070701000A47C7000081A40000000000000000000000015B64B4DF000000D3000000FD0000000200000000000000000000005700000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/doc/source/service/preferences.rst.. (c) Copyright 2016 Hewlett Packard Enterprise Development LP (c) Copyright 2017 SUSE LLC Preferences Service ------------------- .. autoclass:: bll.plugins.preferences_service.PreferencesSvc :members: 070701000A47CB000081A40000000000000000000000015B64B4DF000000CE000000FD0000000200000000000000000000005600000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/doc/source/service/user_group.rst.. (c) Copyright 2016 Hewlett Packard Enterprise Development LP (c) Copyright 2017 SUSE LLC User Group Service ------------------ .. autoclass:: bll.plugins.user_group_service.UserGroupSvc :members: 07070100081050000081A40000000000000000000000015B64B4DF00000179000000FD0000000200000000000000000000004C00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/doc/source/services.rst.. (c) Copyright 2016 Hewlett Packard Enterprise Development LP (c) Copyright 2017 SUSE LLC Plugins ======= .. This includes all of the entries under the services subdirectory. This yields each service's documentation on its own html page, rather than joining them altogether in one massively unreadable page .. toctree:: :maxdepth: 2 :glob: service/* 0707010008105D000081A40000000000000000000000015B64B4DF000003DE000000FD0000000200000000000000000000004500000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/requirements.txt# The order of packages is significant, because pip processes them in the order # of appearance. Changing the order has an impact on the overall integration # process, which may cause wedges in the gate later. # (c) Copyright 2015-2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017-2018 SUSE LLC pbr!=2.1.0,>=2.0.0 # Apache-2.0 Mako>=0.4.0 # MIT WebOb>=1.7.1 # MIT requests>=2.14.2 # Apache-2.0 pecan!=1.0.2,!=1.0.3,!=1.0.4,!=1.2,>=1.0.0 # BSD argparse==1.2.1 simplegeneric==0.8.1 Pykka==1.2.0 oslo.serialization!=2.19.1,>=1.10.0 # Apache-2.0 oslo.i18n!=3.15.2,>=2.1.0 # Apache-2.0 oslo.utils>=3.20.0 # Apache-2.0 PyMySQL>=0.7.6 # MIT License dogpile.cache>=0.6.2 # BSD pyOpenSSL>=0.14 # Apache-2.0 stevedore>=1.20.0 # Apache-2.0 python-keystoneclient>=3.8.0 # Apache-2.0 python-neutronclient>=6.3.0 # Apache-2.0 python-novaclient>=9.0.0 # Apache-2.0 python-cinderclient>=3.1.0 # Apache-2.0 python-monascaclient>=1.7.0 # Apache-2.0 python-ironicclient>=1.14.0 # Apache-2.0 07070100080FEF000081ED0000000000000000000000015B64B4DF000020D9000000FD0000000200000000000000000000004100000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/run_tests.sh#!/bin/bash # (c) Copyright 2015-2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC function usage { echo echo "Usage: $0 [OPTION]..." echo "Run BLL's test suite(s)" echo "" echo " -V, --virtual-env Always use virtualenv. Install automatically" echo " if not present" echo " -N, --no-virtual-env Don't use virtualenv. Run tests in local" echo " environment" echo " -f, --functional=s1[,s2..] Functional Tests that require services s1,s2.." echo " -c, --coverage Generate reports using Coverage" echo " -y, --pylint Just run pylint" echo " -p, --pep8 Run Pep8 syntax checker" echo " -q, --quiet Run non-interactively. (Relatively) quiet." echo " Implies -V if -N is not set." echo " --runserver Run the development server" echo " --docs Just build the documentation" echo " --destroy-environment Destroy the environment and exit" echo " -h, --help Print this usage message" echo "" echo "Note: with no options specified, the script will try to run the tests in" echo " a virtual environment, If no virtualenv is found, the script will ask" echo " if you would like to create one. If you prefer to run tests NOT in a" echo " virtual environment, simply pass the -N option." exit } # # DEFAULTS FOR RUN_TESTS.SH # root=`pwd -P` # To run functional tests against keystone when its endpoint runs over https, # do the following as sudo # 1. download the appropriate certificate from the deployer system into the # /usr/local/share/ca-certificates dir on your system. # This certificate is normally static, so you only have to do that once. # # 2. update-ca-certificates # Direct the Requests library to use the installed ssl standard certs REQUESTS_CA_BUNDLE=${REQUESTS_CA_BUNDLE:-/etc/ssl/certs/ca-certificates.crt} export REQUESTS_CA_BUNDLE # Avoid a bunch of unnecessary warnings about bad ssl certs export PYTHONWARNINGS=ignore::Warning set -o errexit venv=$root/.venv venv_env_version=$venv/environment with_venv=tools/with_venv.sh included_dirs="bll" mkdir -p log always_venv=0 command_wrapper="" destroy=0 just_pylint=0 just_docs=0 just_pep8=0 never_venv=0 quiet=0 runserver=0 testargs="" with_coverage=0 function process_options { # Verify, normalize, sanitize arguments ARGS=$(getopt -o hVNf:cypq -l help,virtual-env,no-virtual-env,functional:,coverage,pylint,pep8,quiet,runserver,docs,destroy-environment -- "$@") || usage eval set -- "$ARGS" while true ; do case "$1" in -h|--help) usage;; -V|--virtual-env) always_venv=1; never_venv=0;; -N|--no-virtual-env) always_venv=0; never_venv=1;; -f|--functional) shift; export functional=$1 ;; -c|--coverage) with_coverage=1;; -y|--pylint) just_pylint=1;; -p|--pep8) just_pep8=1;; -q|--quiet) quiet=1;; --runserver) runserver=1;; --docs) just_docs=1;; --destroy-environment) destroy=1;; --) shift; break;; esac shift done testargs="$@" } function run_server { cd ${root} echo "Starting pecan development server ..." ${command_wrapper} python ${root}/setup.py develop if [[ $testargs && -f $testargs ]] ; then conf_file=$testargs elif [[ $BLL_CONF_OVERRIDE ]] ; then conf_file=$BLL_CONF_OVERRIDE else conf_file=${root}/tests/config.py fi ${command_wrapper} pecan serve ${conf_file} echo "Server stopped." } function destroy_venv { echo "Cleaning environment..." echo "Removing virtualenv..." rm -rf $venv echo "Virtualenv removed." echo "Environment cleaned." } # Determine the version of the environment programatically. For # simplicity, it is the concatenation of the requiments files function get_version { cat requirements.txt test-requirements.txt } function environment_check { echo "Checking environment." if get_version | diff -q $venv_env_version - &> /dev/null ; then # If the environment exists and is up-to-date then set our variables command_wrapper="${root}/${with_venv}" echo "Environment is up to date." return 0 fi if [ $always_venv -eq 1 ]; then install_venv --venv=${venv} else if [ ! -e ${venv}/bin ]; then echo -e "Environment not found. Install? (Y/n) \c" else echo -e "Your environment appears to be out of date. Update? (Y/n) \c" fi read update_env if [ "x$update_env" = "xY" -o "x$update_env" = "x" -o "x$update_env" = "xy" ]; then install_venv --venv=${venv} else # Set our command wrapper anyway. command_wrapper="${root}/${with_venv}" fi fi } function sanity_check { # Anything that should be determined prior to running the tests, server, etc. # Don't sanity-check anything environment-related in -N flag is set if [ $never_venv -eq 0 ]; then if [ ! -e ${venv} ]; then echo "Virtualenv not found at $venv. Did install_venv.py succeed?" exit 1 fi fi # Remove .pyc files. This is sanity checking because they can linger # after old files are deleted. find . -name "*.pyc" -exec rm -rf {} \; } function install_venv { # Install with install_venv.py export PIP_DOWNLOAD_CACHE=${PIP_DOWNLOAD_CACHE-/tmp/.pip_download_cache} export PIP_USE_MIRRORS=${PIP_USE_MIRRORS:=true} if [ $quiet -eq 1 ]; then export PIP_NO_INPUT=${PIP_NO_INPUT:=true} fi echo "Fetching new src packages..." rm -rf $venv/src python tools/install_venv.py --venv=${venv} command_wrapper="$root/${with_venv}" # Make sure it worked and record the environment version # sanity_check chmod -R 754 $venv get_version > $venv_env_version } function run_pep8 { sanity_check ${command_wrapper} flake8 bll tests } function run_pylint { echo "Running pylint ..." PYTHONPATH=$root ${command_wrapper} pylint --rcfile=.pylintrc $included_dirs > pylint.txt || true CODE=$? grep Global -A2 pylint.txt if [ $CODE -lt 32 ]; then echo "Completed successfully." exit 0 else echo "Completed with problems." exit $CODE fi } function run_sphinx { echo "Building sphinx..." ${command_wrapper} sphinx-build -b html doc/source doc/build echo "Build complete." } function run_tests { sanity_check cd ${root} local q="" if [ $quiet -eq 1 ]; then q=-q fi if [ -n "$testargs" ] ; then s=-s fi if [ $with_coverage -eq 1 ]; then ${command_wrapper} coverage2 erase ${command_wrapper} coverage2 run $root/setup.py test $q $s $testargs else ${command_wrapper} python $root/setup.py test $q $s $testargs fi API_RESULT=$? if [ $with_coverage -eq 1 ]; then echo "Generating coverage reports" ${command_wrapper} coverage2 xml -i --omit="/usr*,setup.py,*egg*,${venv}/*,tests/*,examples/*" ${command_wrapper} coverage2 html -i --omit="/usr*,setup.py,*egg*,${venv}/*,tests/*,examples/*" -d reports fi if [ $API_RESULT -eq 0 ]; then echo "Tests completed successfully." else echo "Tests failed." fi exit $API_RESULT } # ---------PREPARE THE ENVIRONMENT------------ # # PROCESS ARGUMENTS, OVERRIDE DEFAULTS process_options "$@" # If destroy is set, just blow it away and exit. if [ $destroy -eq 1 ]; then destroy_venv exit 0 fi if [ $quiet -eq 1 ] && [ $never_venv -eq 0 ] && [ $always_venv -eq 0 ] then always_venv=1 fi # Ignore all of this if the -N flag was set if [ $never_venv -eq 0 ]; then environment_check fi # ---------EXERCISE THE CODE------------ # # Pylint if [ $just_pylint -eq 1 ]; then run_pylint exit $? fi # Build the docs if [ $just_docs -eq 1 ]; then run_sphinx exit $? fi # Pep8 if [ $just_pep8 -eq 1 ]; then run_pep8 exit $? fi # pecan development server if [ $runserver -eq 1 ]; then run_server exit $? fi # Full test suite run_tests || exit 07070100080FF2000081A40000000000000000000000015B64B4DF000000FD000000FD0000000200000000000000000000003E00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/setup.cfg# (c) Copyright 2015-2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC [metadata] name = bll [files] packages = bll [nosetests] match=^test where=bll nocapture=1 cover-package=bll cover-erase=1 [coverage:run] source = bll 07070100081057000081ED0000000000000000000000015B64B4DF00000968000000FD0000000200000000000000000000003D00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/setup.py#!/usr/bin/env python # # (c) Copyright 2015-2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017-2018 SUSE LLC # try: from setuptools import setup, find_packages except ImportError: from ez_setup import use_setuptools use_setuptools() from setuptools import setup, find_packages # Convenience function creating long strings below def entry(plugin_name, class_name, file_name=None): return "%(plugin)s = bll.plugins.%(file)s:%(class)s" % \ {"plugin": plugin_name, "file": file_name or (plugin_name + "_service"), "class": class_name} setup( name='ardana-opsconsole-server', version='1.0', author='SUSE LLC', url='https://github.com/ArdanaCLM', packages=find_packages(exclude=['tests', 'tests.*', 'stubs', 'stubs.*']), license='Apache-2.0', include_package_data=True, scripts=[], description='Business Logic Layer for the Operations Console', # The following are the first-order dependencies, excluding # the openstack dependencies that are installed from source. The # entries here are used by the wheel build (pip wheel .) that is used # by the Ardana build. install_requires=[ 'pecan', 'stevedore', 'PyMySQL', 'Pykka', 'argparse', 'WebOb', 'requests', 'dogpile.cache', 'pyOpenSSL>=0.15.1' ], classifiers=[ 'Environment :: OpenStack', 'Intended Audience :: Information Technology', 'Intended Audience :: System Administrators', 'License :: Other/Proprietary License', 'Operating System :: POSIX :: Linux', 'Programming Language :: Python', 'Programming Language :: Python :: 2', 'Programming Language :: Python :: 2.7', ], test_suite='tests', zip_safe=False, entry_points={ 'bll.plugins': [ entry("catalog", "CatalogSvc"), entry("cinder", "CinderSvc"), entry("compute", "ComputeSvc"), entry("eula", "EulaSvc"), entry("ardana", "ArdSvc"), entry("ironic", "IronicSvc"), entry("monitor", "MonitorSvc"), entry("nova", "NovaSvc"), entry("objectstorage_summary", "ObjectStorageSummarySvc"), entry("preferences", "PreferencesSvc"), entry("user_group", "UserGroupSvc"), ], }, ) 070701000DFF75000041ED0000000000000000000000045B64B4DF00000000000000FD0000000200000000000000000000003A00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/stubs070701000DFF7C000081A40000000000000000000000015B64B4DF00000000000000FD0000000200000000000000000000004600000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/stubs/__init__.py070701000DFF7D000041ED0000000000000000000000025B64B4DF00000000000000FD0000000200000000000000000000004100000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/stubs/common070701000DFF7E000081A40000000000000000000000015B64B4DF00000000000000FD0000000200000000000000000000004D00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/stubs/common/__init__.py070701000DFF7F000081A40000000000000000000000015B64B4DF00000279000000FD0000000200000000000000000000005400000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/stubs/common/dict_job_status.py# (c) Copyright 2016-2017 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC from bll import api class DictStatus(object): """ Implementation of job status using a dictionary in memory. This is only suitable for unit/functional testing in a non-clustered environment. """ status_dict = {} def update_job_status(self, txn_id, status): DictStatus.status_dict[txn_id] = status def get_job_status(self, txn_id): if txn_id in DictStatus.status_dict: return DictStatus.status_dict[txn_id] else: return {api.STATUS: api.STATUS_NOT_FOUND} 070701000DFF76000041ED0000000000000000000000025B64B4DF00000000000000FD0000000200000000000000000000004200000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/stubs/plugins070701000DFF77000081A40000000000000000000000015B64B4DF00000000000000FD0000000200000000000000000000004E00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/stubs/plugins/__init__.py070701000DFF79000081A40000000000000000000000015B64B4DF000008A5000000FD0000000200000000000000000000005700000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/stubs/plugins/composite_service.py# (c) Copyright 2015-2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC from bll.plugins import service from bll import api from bll.api.request import BllRequest class CompositeSvc(service.SvcBase): """ Service with sync calls that calls sync calls in other plugins """ @service.expose() def composite(self): foo = self.call_service(target='general', operation='echo', data={ 'message': 'foo'}) bar = self.call_service(target='general', operation='echo', data={ 'message': 'bar'}) return [foo, bar] @service.expose() def fail(self): return self.call_service(target='general', operation='failhandle') class CompositeAsyncSvc(service.SvcBase): def handle(self): self.response[api.STATUS] = api.STATUS_INPROGRESS self.response[api.PROGRESS] = {api.PERCENT_COMPLETE: 0} return self.response def complete(self): """ Async call that calls async calls in other plugins """ if self.operation == 'progress': request = BllRequest(target="general", operation='progress', data={'num_pauses': 2}) elif self.operation == 'fail': request = BllRequest(target="general", operation="failcomplete") self.response[api.DATA] = self.call_service_async(request, polling_interval=0.1) self.response[api.PERCENT_COMPLETE] = dict(percentComplete=100) self.response.complete() return self.response class CompositeAsyncExposeSvc(service.SvcBase): """ Async call that calls async calls in other plugins, all using @expose """ @service.expose(is_long=True) def go(self, validate): if validate: return if self.operation == 'progress': request = BllRequest(target="general", operation='progress', data={'num_pauses': 2}) elif self.operation == 'fail': request = BllRequest(target="general", operation="failcomplete") return self.call_service_async(request, polling_interval=0.1) 070701000DFF7B000081A40000000000000000000000015B64B4DF000009F4000000FD0000000200000000000000000000005400000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/stubs/plugins/expose_service.py# (c) Copyright 2015-2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC from builtins import range import time from bll.plugins import service from bll.common.util import get_val from bll import api class ExposeSvc(service.SvcBase): """ Stub class that tests the service dispatching via the @expose decorator """ @service.expose('valid_op') def do_op(self): return 'ok' def not_exposed(self): pass @service.expose(action='act') def do_action(self): return 'action' @service.expose('op1') @service.expose('op2') def do_multi(self): return 'multi' @service.expose('slow_op', is_long=True) def do_slow_op(self): """ This is an example of a long-running operation that expects no parameters. It should be called once: during the complete() function. """ self.response[api.PROGRESS] = {api.PERCENT_COMPLETE: 100} self.put_resource(self.request.txn_id, self.response) return self.response @service.expose('progress', is_long=True) def progress(self, validate): """ This is an example of a long-running operation that expects a parameter. It should be called twice: once during the handle() function with the parameter set to True, and once during the complete() with the parameter set to False. Note that this function behaves the same as the ProgressSvc example, and the unit test code that calls it is nearly identical, underscoring the fact that the resulting behavior is the same as the legacy handle/complete. """ if validate: # Validate the input parameters and return a suggested # poll interval self.pause_sec = get_val(self.request, "data.pause_sec", 0.1) self.num_pauses = get_val(self.request, "data.num_pauses", 100) return self.pause_sec else: for count in range(1, self.num_pauses + 1): time.sleep(self.pause_sec) progress = (100 * count) / self.num_pauses self.response[api.PROGRESS] = {api.PERCENT_COMPLETE: progress} self.put_resource(self.request.txn_id, self.response) self.response.complete() return self.response @service.expose('data_in_response') def data_in_response(self): self.response[api.DATA] = 'blah' self.response.complete() # return nothing 070701000DFF7A000081A40000000000000000000000015B64B4DF0000076E000000FD0000000200000000000000000000005500000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/stubs/plugins/general_service.py# (c) Copyright 2016-2017 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC from builtins import range from bll import api from bll.plugins import service from bll.common import util import time class GeneralSvc(service.SvcBase): """ General-purpose service class for testing. """ @service.expose() def null(self): return [] @service.expose() def echo(self): return self.data.get('message') @service.expose(is_long=True) def echo_slow(self): return self.data.get('message') @service.expose() def failhandle(self): raise Exception('Intentional exception in handle') @service.expose(is_long=True) def failcomplete(self): raise Exception('Intentional exception in complete') @service.expose(is_long=True) def errorcomplete(self): self.response.error('some error happened') return self.response @service.expose(is_long=True) def progress(self, validate): # Long running process that sleeps and posts updates if validate: self.pause_sec = util.get_val(self.request, "data.pause_sec", 0.1) self.num_pauses = util.get_val(self.request, "data.num_pauses", 100) return self.pause_sec else: for count in range(1, self.num_pauses + 1): time.sleep(self.pause_sec) progress = (100 * count) / self.num_pauses self.response[api.PROGRESS] = {api.PERCENT_COMPLETE: progress} self.put_resource(self.request.txn_id, self.response) self.response.complete() return self.response class UnavailableSvc(service.SvcBase): """ Class for testing the catalog service. """ @classmethod def needs_services(cls): return ['SomeMissingService'] 070701000DFF78000081A40000000000000000000000015B64B4DF00000556000000FD0000000200000000000000000000004B00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/stubs/plugins/setup.py# (c) Copyright 2015-2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC from setuptools import setup, find_packages setup( name='bll-services-stubs', version='1.0', description='Stub services to support bll testing', classifiers=['Development Status :: 3 - Alpha', 'License :: OSI Approved :: Apache Software License', 'Programming Language :: Python', 'Programming Language :: Python :: 2', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3', 'Programming Language :: Python :: 3.2', 'Programming Language :: Python :: 3.3', 'Intended Audience :: Developers', 'Environment :: Console', ], platforms=['Any'], scripts=[], packages=find_packages(), include_package_data=True, entry_points={'bll.plugins': [ 'composite = stubs.plugins.composite_service:CompositeSvc', 'composite-async = stubs.plugins.composite_service:' 'CompositeAsyncSvc', 'general = stubs.plugins.general_service:GeneralSvc', 'expose = stubs.plugins.expose_service:ExposeSvc', 'unavailable = stubs.plugins.general_service:UnavailableSvc', ], }, zip_safe=False, ) 07070100080FF1000081A40000000000000000000000015B64B4DF000001E4000000FD0000000200000000000000000000004A00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/test-requirements.txt# The order of packages is significant, because pip processes them in the order # of appearance. Changing the order has an impact on the overall integration # process, which may cause wedges in the gate later. # (c) Copyright 2015-2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017-2018 SUSE LLC #Testing Requirements WebTest>=2.0 # MIT coverage!=4.4,>=4.0 # Apache-2.0 WSME>=0.8 # MIT pylint>=1.0.0 flake8>=2.2.2 mock>=2.0 # BSD sphinx>=1.6.2 # BSD -e stubs/plugins 070701000A47CE000041ED0000000000000000000000055B64B4DF00000000000000FD0000000200000000000000000000003A00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/tests070701000A47E7000081A40000000000000000000000015B64B4DF00000000000000FD0000000200000000000000000000004600000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/tests/__init__.py070701000A47DF000041ED0000000000000000000000035B64B4DF00000000000000FD0000000200000000000000000000003E00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/tests/api070701000A47E4000081A40000000000000000000000015B64B4DF00000000000000FD0000000200000000000000000000004A00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/tests/api/__init__.py070701000A47E0000041ED0000000000000000000000025B64B4DF00000000000000FD0000000200000000000000000000004900000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/tests/api/controller070701000A47E1000081A40000000000000000000000015B64B4DF00000000000000FD0000000200000000000000000000005500000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/tests/api/controller/__init__.py070701000A47E2000081A40000000000000000000000015B64B4DF000018CC000000FD0000000200000000000000000000005C00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/tests/api/controller/test_controller.py# (c) Copyright 2015-2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC from datetime import datetime import json import logging import mock import time from bll import api from bll.api.controllers import app_controller from bll.api.controllers.v1 import V1 from bll.common.job_status import get_job_status from tests.util import TestCase, log_level @mock.patch.object(app_controller, 'response') @mock.patch.object(app_controller, 'request') class Test(TestCase): def test_post_no_txn_id(self, _mock_request, _mock_response): _mock_request.body = json.dumps({ api.TARGET: 'general', api.DATA: { api.OPERATION: 'null', } }) reply = app_controller.AppController().post() self.assertEqual(reply[api.STATUS], api.COMPLETE) def test_status_update_no_txn_id(self, _mock_request, _mock_response): _mock_request.body = json.dumps({ api.TARGET: 'general', api.DATA: { api.OPERATION: 'null', }, api.JOB_STATUS_REQUEST: True }) with log_level(logging.CRITICAL, 'bll'): reply = app_controller.AppController().post() self.assertEqual(reply[api.STATUS], api.STATUS_ERROR) self.assertIn("No txn_id", reply[api.DATA][0][api.DATA]) self.assertTrue(_mock_response.status, 400) def test_post_request_fail(self, _mock_request, _mock_response): _mock_request.body = json.dumps({api.TARGET: 'general', api.DATA: {api.OPERATION: 'failhandle'}}) # Suppress the expected exception message with log_level(logging.CRITICAL, 'bll'): reply = app_controller.AppController().post() self.assertEqual(reply[api.STATUS], api.STATUS_ERROR) self.assertTrue(_mock_response.status, 400) def test_post_complete_fail(self, _mock_request, _mock_response): _mock_request.body = json.dumps({api.TARGET: 'general', api.DATA: {api.OPERATION: 'failcomplete'}}) # Suppress the expected exception message from service with log_level(logging.CRITICAL, 'bll.plugins.service'): reply = app_controller.AppController().post() time.sleep(0.1) txn_id = reply.get(api.TXN_ID) reply = get_job_status(txn_id) self.assertEqual(reply[api.STATUS], 'error') def test_post_complete_error(self, _mock_request, _mock_response): _mock_request.body = json.dumps({api.TARGET: 'general', api.DATA: {api.OPERATION: 'errorcomplete'}}) # Suppress the expected exception message from service with log_level(logging.CRITICAL, 'bll.plugins.service'): reply = app_controller.AppController().post() time.sleep(0.1) txn_id = reply.get(api.TXN_ID) reply = get_job_status(txn_id) self.assertEqual(reply[api.STATUS], 'error') self.assertEqual(reply[api.DATA][0][api.DATA], 'some error happened') class TestV1(TestCase): @classmethod def setUpClass(cls): super(TestCase, cls).setUpClass() cls.load_test_app() def testIndex(self): assert self.app.get('/').status_int == 200 def testV1Index(self): assert self.app.get('/v1/').status_int == 200 @mock.patch('bll.api.controllers.v1.login', return_value=type('X', (object,), dict(auth_token='foo', expires=datetime.utcnow()))) def testLogin(self, mock_login): body = {'username': 'user', 'password': 'pass'} response = self.app.post_json('/v1/auth_token', body) self.assertEqual(200, response.status_code) @mock.patch('bll.api.controllers.v1.login', side_effect=Exception('foo')) def testFailLogin(self, _): body = {'username': 'user', 'password': 'pass'} response = self.app.post_json('/v1/auth_token', body, expect_errors=True) self.assertEqual(401, response.status_code) def testBypassPermissions(self): # Create a request object for eula, which bypasses permission checks request = type('Request', (object,), dict(body='{"target": "eula"}')) with mock.patch('bll.api.controllers.v1.request', request): self.assertTrue(V1.check_permissions()) def testPermissionsWithoutToken(self): # Create a request object without a token request = type('Request', (object,), dict(body='{"target": "foo"}', headers='{}')) with mock.patch('bll.api.controllers.v1.request', request): self.assertFalse(V1.check_permissions()) @mock.patch('bll.api.controllers.v1.validate', return_value=True) def testPermissionsWithToken(self, _): # Create a request object with a token request = type('Request', (object,), dict(body='{"target": "foo"}', headers={"X-Auth-Token": "sometoken"})) with mock.patch('bll.api.controllers.v1.request', request): self.assertTrue(V1.check_permissions()) @mock.patch('bll.api.controllers.v1.validate', return_value=True) def testBackwardCompat(self, _): # Create an old token blob as a json string headers = """ {"management_appliance" : {"tokens": [{"auth_token" : "blah" }]}} """ # Create a bogus request object request = type('Request', (object,), dict(body='{"target": "plugins"}', headers={"X-Auth-Token": headers})) with mock.patch('bll.api.controllers.v1.request', request): self.assertTrue(V1.check_permissions()) @mock.patch('bll.api.controllers.v1.validate', return_value=True) def test_missing_service(self, _): # Suppress the expected exception message from service body = {'target': 'bogus-service'} response = self.app.post_json('/v1/bll', body, expect_errors=True) self.assertEqual(401, response.status_code) 070701000A47E5000081A40000000000000000000000015B64B4DF000017BC000000FD0000000200000000000000000000004B00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/tests/api/test_auth.py# # (c) Copyright 2015-2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC # from bll.api.auth_token import TokenHelpers from bll.common.exception import BllAuthenticationFailedException from tests.util import TestCase, functional, create_user, delete_user import bll.api.auth_token as auth_token @functional('keystone') class TestAuth(TestCase): def setUp(self): self.user = None def tearDown(self): if self.user: delete_user(self.user) def login(self): (self.user, password) = create_user({'admin': 'admin'}, {'Default': 'admin'}) auth_ref = auth_token.login(self.user.name, password) self.assertIsNotNone(auth_ref) self.assertIsNotNone(auth_ref.auth_token) self.assertTrue(auth_token.validate(auth_ref.auth_token)) return auth_ref def test_valid_credentials(self): self.login() def test_invalid_credentials(self): self.assertRaises(Exception, auth_token.login, 'invalid', 'invalid') def test_get_appropriate_auth_ref(self): auth_ref = self.login() new_auth_ref = auth_token.get_appropriate_auth_ref(auth_ref.auth_token) self.assertEqual(auth_ref.project_name, new_auth_ref.project_name) def test_get_auth_ref(self): auth_ref = self.login() token = auth_ref.auth_token ref = auth_token.get_appropriate_auth_ref(token) self.assertIsNotNone(ref) self.assertIsNotNone(ref.auth_token) self.assertEqual('admin', ref.project_name) def test_get_auth_ref_invalid_project(self): auth_ref = self.login() token = auth_ref.auth_token self.assertRaises(Exception, auth_token._get_auth_ref, token, 'invalid') def test_get_auth_ref_invalid_token(self): self.assertRaises(Exception, auth_token._get_auth_ref, 'invalid', 'admin') def test_get_auth_url(self): # Just a sanity test of the function self.assertIsNotNone(auth_token.get_auth_url()) def test_helper_token(self): auth_ref = self.login() helper = TokenHelpers(auth_ref.auth_token) self.assertEqual(auth_ref.auth_token, helper.get_user_token()) def test_helper_service_endpoint(self): auth_ref = self.login() helper = TokenHelpers(auth_ref.auth_token) self.assertIsNotNone(helper.get_service_endpoint("identity")) def test_login_fails_without_project_or_domain(self): (self.user, password) = create_user() with self.assertRaisesRegexp(BllAuthenticationFailedException, 'access to the admin project'): auth_token.login(self.user.name, password) def test_login_fails_demo_project_admin(self): # Admin role on demo project, but missing the admin project (self.user, password) = create_user({'demo': 'admin'}, {'Default': 'admin'}) with self.assertRaisesRegexp(BllAuthenticationFailedException, 'access to the admin project'): auth_token.login(self.user.name, password) def test_login_fails_admin_project(self): # Admin role on admin project, but missing the domain admin (self.user, password) = create_user({'admin': 'admin'}) with self.assertRaisesRegexp(BllAuthenticationFailedException, 'not authorized on the .* domain'): auth_token.login(self.user.name, password) def test_login_fails_multiple_admin_projects(self): # Admin role on both projects, but missing the domain admin (self.user, password) = create_user({'admin': 'admin', 'demo': 'admin'}) with self.assertRaisesRegexp(BllAuthenticationFailedException, 'not authorized on the .* domain'): auth_token.login(self.user.name, password) def test_login_fails_domain_admin(self): # Domain admin, but lacking access to admin project (self.user, password) = create_user( {'demo': 'admin'}, {'Default': 'admin'}) with self.assertRaisesRegexp(BllAuthenticationFailedException, 'access to the admin project'): auth_token.login(self.user.name, password) def test_login_fails_domain_project_admin(self): # Domain admin, and demo project admin, but lacking access to admin # project (self.user, password) = create_user({'demo': 'admin'}, {'Default': 'admin'}) with self.assertRaisesRegexp(BllAuthenticationFailedException, 'access to the admin project'): auth_token.login(self.user.name, password) def test_login_other_admin(self): # Domain admin, and demo project admin, and monasca-user role # on admin project. This should succeed (self.user, password) = create_user({'admin': 'monasca-user', 'demo': 'admin'}, {'Default': 'admin'}) auth_ref = auth_token.login(self.user.name, password) token = auth_ref.auth_token ref = auth_token.get_appropriate_auth_ref(token) self.assertIsNotNone(ref) self.assertIsNotNone(ref.auth_token) self.assertEqual('demo', ref.project_name) def test_login_fails_domain_member(self): # Admin role on admin project, domain member, but missing the domain # admin (self.user, password) = create_user({'admin': 'admin'}, {'Default': '_member_'}) with self.assertRaisesRegexp(BllAuthenticationFailedException, 'not an admin of the default domain'): auth_token.login(self.user.name, password) 070701000A47E3000081A40000000000000000000000015B64B4DF00000FDA000000FD0000000200000000000000000000004E00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/tests/api/test_request.py# (c) Copyright 2015-2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC from bll import api from tests import util from bll.api.request import BllRequest class Test(util.TestCase): def test_chained_creation(self): req1 = BllRequest(target=util.randomword(), operation=util.randomword()) req2 = BllRequest(req1) self.assertEquals(req1, req2) def test_creation_from_dict(self): req1 = dict(target=util.randomword(), operation=util.randomword()) req2 = BllRequest(req1) req3 = BllRequest(req2) self.assertEquals(req2, req3) def test_overrides(self): # Test that explicitly supplied values override those in the # request parameter of the BllRequest constructor req1 = BllRequest(target=util.randomword(), auth_token=util.randomword(), operation=util.randomword(), action=util.randomword(), data=util.randomdict()) target = util.randomword() operation = util.randomword() action = util.randomword() auth_token = util.randomword() req2 = BllRequest(request=req1, target=target, operation=operation, action=action, auth_token=auth_token) self.assertEquals(req2['action'], action) self.assertEquals(req2['target'], target) self.assertEquals(req2['auth_token'], auth_token) self.assertEquals(req2['data']['operation'], operation) def test_data_remains_gone_when_none_supplied(self): # Verify that when neither 'operation' nor 'data' are supplied, that # the resulting request has no 'data' key req1 = BllRequest(target=util.randomword(), action=util.randomword()) self.assertFalse(req1.get('data')) def test_flattening(self): # Verify that we get the same result whether creating from a # dictionary, individual fields, or a nested data element txn_id = util.randomhex() target = util.randomword() op = util.randomword() d = util.randomdict() req1 = BllRequest(dict(target=target, foo="baz", txn_id=txn_id, operation=op, bar=d)) req2 = BllRequest(target=target, foo="baz", txn_id=txn_id, operation=op, bar=d) req3 = BllRequest(target=target, txn_id=txn_id, data={'operation': op, 'foo': 'baz', 'bar': d}) self.assertDictEqual(req1, req2) self.assertDictEqual(req2, req3) self.assertIn("operation", req1['data']) self.assertIn("foo", req1['data']) self.assertIn("bar", req1['data']) self.assertNotIn("target", req1['data']) self.assertNotIn("txn_id", req1['data']) def test_doubly_nested_data(self): target = util.randomword() d = util.randomdict() req = BllRequest(target=target, data={'data': d}) # Make sure that the doubly nested data got populated correctly self.assertDictEqual(d, req['data']['data']) def test_get_data(self): # Verify that get_data returns all non reserved fields correctly req = BllRequest(target=util.randomword(), action="GET", foo=util.randomword(), txn_id=util.randomhex(), auth_token=util.randomhex(), operation=util.randomword(), version="1") data = req.get_data() self.assertNotIn("action", data) self.assertNotIn("target", data) self.assertNotIn("txn_id", data) self.assertNotIn("auth_token", data) self.assertNotIn("region", data) self.assertNotIn("data", data) self.assertNotIn(api.VERSION, data) self.assertNotIn("operation", data) self.assertIn("foo", data) 070701000A47E8000041ED0000000000000000000000025B64B4DF00000000000000FD0000000200000000000000000000004100000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/tests/common070701000A47EA000081A40000000000000000000000015B64B4DF00000000000000FD0000000200000000000000000000004D00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/tests/common/__init__.py070701000A47E9000081A40000000000000000000000015B64B4DF00002BCC000000FD0000000200000000000000000000004E00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/tests/common/test_util.py# (c) Copyright 2015-2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017-2018 SUSE LLC import copy from bll.common import util from tests import util as test_util from bll.common.util import scrub_passwords from simplejson import loads from bll.api.request import BllRequest from pecan import conf FTI = { "http_proxy_settings": { "no_proxy": "10.*,20.*,localhost", "https_proxy": "secure-proxy", "http_proxy": "open-proxy" }, "appl_endpts": { "ent_appl": { "DCM": { "vip": "16.1.1.127", "hostname": "mgr.mycloud.com" }, "CAN": { "public_ip": "16.1.1.129", "vip": "16.1.1.128", "hostname": "mgr.mycloud.com" } }, "mgmt_appl": { "DCM": { "vip": "16.1.1.123", "hostname": "mgr.mycloud.com" } }, "cc_appl": { "DCM": { "vip": "16.1.1.124", "hostname": "mgr.mycloud.com" }, "CAN": { "public_ip": "16.1.1.126", "vip": "16.1.1.125", "hostname": "mgr.mycloud.com" } }, "mon_appl": { "DCM": { "vip": "16.1.1.129", "hostname": "monitor.mycloud.com" } } }, "locale_settings": { "locale": "lunar", "time_zone": "twenty-fifth-hour" }, "time_server": ["time-server-1", "time-server-2"], "activate_ent_appl": False, "glance_disk_size": 512, "migration": False, "images_settings": { "appl_images": [{ "avm_name": "mgmt-controller", "avm_type": "mgmt-controller", "avm_role": "CloudController", "avm_image": "foundation" }, { "avm_name": "enterprise1-controller", "avm_type": "enterprise1", "avm_role": "EnterpriseController", "avm_image": "enterprise" }]}, "images_path": "\/legacy" } blanked_password = '****' class Test(test_util.TestCase): def test_merge_empty(self): fti_settings = {} proxy = { "http_proxy_settings": { "no_proxy": "10.*,20.*,localhost", "https_proxy": "secure-proxy", "http_proxy": "open-proxy" }, } merged = util.deepMerge(fti_settings, proxy) self.assertDictEqual(merged, proxy) def test_merge_full(self): fti_settings = copy.deepcopy(FTI) proxy = { "http_proxy_settings": { "no_proxy": "10.*,20.*,foo", "https_proxy": "foo-proxy", "http_proxy": "open-proxy" }, } merged = util.deepMerge(fti_settings, proxy) result = fti_settings result['http_proxy_settings'] = proxy['http_proxy_settings'] self.assertDictEqual(merged, result) def test_merge_timeserver(self): fti_settings = copy.deepcopy(FTI) time = { "time_server": [ "time-server-4", "time-server-25", "tm-45" ], } merged = util.deepMerge(fti_settings, time) self.assertTrue(len(merged['time_server']) == 3) result = fti_settings result['time_server'] = time['time_server'] self.assertDictEqual(merged, result) fti_settings = copy.deepcopy(FTI) time = { "time_server": [], } merged = util.deepMerge(fti_settings, time) self.assertTrue(len(merged['time_server']) == 0) result = fti_settings result['time_server'] = time['time_server'] self.assertDictEqual(merged, result) def test_merge_timezone(self): fti_settings = copy.deepcopy(FTI) time = { "locale_settings": { "time_zone": "Ireland" }, } merged = util.deepMerge(fti_settings, time) result = fti_settings result['locale_settings']['time_zone'] =\ time["locale_settings"]["time_zone"] self.assertDictEqual(merged, result) def test_scrub_passwords_single(self): test_data = { 'blah': 'something', 'password': 'password', 'pre_password': 'password', 'password_post': 'password' } result = loads(scrub_passwords(test_data)) self.assertEquals(result['password'], blanked_password) self.assertEquals(result['pre_password'], blanked_password) self.assertEquals(result['password_post'], blanked_password) self.assertEquals(result['blah'], 'something') def test_scrub_passwords_deep(self): test_data = { 'blah': 'something', 'password': 'password', 'lvl1': { 'a': 'a', 'password': 'password', 'lvl2': { 'b': 'b', 'password': 'password', 'lvl3': { 'c': 'c', 'password': 'password' } } } } result = loads(scrub_passwords(test_data)) self.assertEquals(result['password'], blanked_password) self.assertEquals(result['lvl1']['password'], blanked_password) self.assertEquals(result['lvl1']['lvl2']['password'], blanked_password) self.assertEquals(result['lvl1']['lvl2']['lvl3']['password'], blanked_password) self.assertEquals(result['lvl1']['lvl2']['lvl3']['c'], 'c') ''' This tests the case where a value is a string containing a JSON object and that JSON object requires password scrubbing ''' def test_scrub_passwords_where_value_is_string(self): body = {'appliance-settings': '{"monasca-controller2": {' '"ip": "192.168.0.35", ' '"keepalive-priority": "98", ' '"keystone-password": "mypassword"}, ' '"fakehost": {' '"ip": "192.168.0.34", ' '"somedict": {' '"testpassword": "mypassword"}}}'} # Valid JSON doesn't allow embedded JSON as a string inside a JSON. # But we do this crazy stuff in our code, so let's make sure this # case still clears for passwords result = scrub_passwords(body) self.assertTrue('mypassword' not in result) def test_scrub_passwords_none(self): result = scrub_passwords(None) self.assertIsNone(result) def test_scrub_passwords_json_in_a_string(self): test_str = '{"ma1": {' \ '"ip": "192.168.0.35", ' \ '"keepalive-priority": "98", ' \ '"keystone-password": "mypassword"}, ' \ '"fakehost": {' \ '"ip": "192.168.0.34", ' \ '"somedict": {' \ '"testpassword": "mypassword"}}}' result = loads(scrub_passwords(test_str)) self.assertEquals(result['ma1']['keystone-password'], blanked_password) self.assertEquals(result['fakehost']['somedict']['testpassword'], blanked_password) ''' To test where people put a dictionary as the only element in an array. ''' def test_scrub_passwords_dict_in_list_with_unicode(self): test_data = [{u'username': u'root', u'name': u'enc1-bay2', u'ip_address': u'192.168.0.71', u'password': u'unset'}] result = loads(scrub_passwords(test_data)) self.assertEquals(result[0]['password'], blanked_password) ''' handle craziness of dumping list of items in rest_wrap.py's method_debug_msg() ''' def test_scrub_passwords_rest_wrap_craziness(self): test_data = [u"action=u'GET'", u"data={u'operation': \ u'get_provider_networks', u'api_version': \ u'v1', u'ops_console_admin_password': u'unset'}", u"txn_id=u'1234'", u"target=u'openstack_network'"] result = scrub_passwords(test_data) self.assertTrue('unset' not in result) def test_scrub_token_element(self): test_data = {'auth_token': 'fake-token', 'txn_id': '8582cdc0-c09a-4bd6-a1a8-67c5af78a1cf', 'target': 'deploy', 'data': {'operation': 'some_operation'}} result = loads(scrub_passwords(test_data)) self.assertEquals(result['auth_token'], '******oken') def test_scrub_token_from_bll_request(self): test_data = {'auth_token': '4d038a60-2409-4636-b99f-280e4dc3c6fe', 'txn_id': '8582cdc0-c09a-4bd6-a1a8-67c5af78a1cf', 'target': 'deploy', 'data': {'operation': 'some_operation'}} result = scrub_passwords(BllRequest(test_data)) self.assertNotIn('4d038a60', result) def test_scrub_short_token_element(self): test_data = {'auth_token': 'FE34', 'txn_id': '8582cdc0-c09a-4bd6-a1a8-67c5af78a1cf', 'target': 'deploy', 'data': {'operation': 'some_operation'}} result = loads(scrub_passwords(test_data)) self.assertEquals(result['auth_token'], 'FE34') def test_scrub_none_token_element(self): test_data = {'auth_token': None, 'txn_id': '8582cdc0-c09a-4bd6-a1a8-67c5af78a1cf', 'target': 'deploy', 'data': {'operation': 'some_operation'}} result = loads(scrub_passwords(test_data)) self.assertEquals(result['auth_token'], 'None') def test_scrub_json_in_a_string_craziness_1(self): test_data = {u'appliance-settings': u'{"monasca-controller2": \ {"ip": "192.168.0.35", "keystone-password": "unset", \ "my_token": "sometoken"}}'} result = scrub_passwords(test_data) self.assertTrue('unset' not in result) self.assertTrue('*****oken' in result) def test_scrub_json_in_a_string_craziness_2(self): test_data = '{"appliance-settings": "{\\"monasca-controller2\\": \ {\\"ip\\": \\"4.3.2.1\\", \ \\"keystone-password\\": \\"unset\\", \ \\"my_token\\": \\"sometoken\\"}}"}' result = scrub_passwords(test_data) self.assertTrue('unset' not in result) self.assertTrue('*****oken' in result) def test_new_txn_id(self): txn_id = util.new_txn_id() self.assertEquals(len(txn_id), 36) # len of uuid strings child1 = util.new_txn_id(txn_id) child2 = util.new_txn_id(txn_id) self.assertNotEqual(child1, child2) # better be different! self.assertEquals(len(txn_id) + 9, len(child1)) self.assertEquals(len(child2), len(child1)) 070701000A47DE000081A40000000000000000000000015B64B4DF00000BB2000000FD0000000200000000000000000000004400000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/tests/config.py# (c) Copyright 2016-2017 Hewlett Packard Enterprise Development LP # (c) Copyright 2017-2018 SUSE LLC import os from bll.hooks import RestHook # Keystone config KEYSTONE_HOST = os.getenv('KEYSTONE_HOST') or '192.168.245.9' KEYSTONE_PROTOCOL = os.getenv('KEYSTONE_PROTOCOL') or 'https' if os.getenv('KEYSTONE_PORT'): KEYSTONE_PORT = int(os.getenv('KEYSTONE_PORT')) else: KEYSTONE_PORT = 5000 # Mysql config DB_HOST = os.getenv('DB_HOST') or 'localhost' DB_NAME = os.getenv('DB_NAME') or 'opsdb' DB_USER = os.getenv('DB_USER') or 'opsconuser' DB_PASSWORD = os.getenv('DB_PASSWORD') or 'opsconpass' if os.getenv('DB_PORT'): DB_PORT = int(os.getenv('DB_PORT')) else: DB_PORT = 3306 LOG_LEVEL = os.getenv('LOG_LEVEL') or 'INFO' insecure = bool(os.getenv('INSECURE')) or False # port for BLL to listen on when using run_tests.sh --runserver PORT = os.getenv('PORT') or '8084' server = { 'port': PORT, 'host': '0.0.0.0', } keystone = { 'private_url': '%s://%s:%d' % (KEYSTONE_PROTOCOL, KEYSTONE_HOST, KEYSTONE_PORT), # 'insecure': True, # Permit insecure https for dev/test 'version': 'v2.0', 'service_tenant': 'services', } db = { 'host': DB_HOST, 'port': DB_PORT, 'database': DB_NAME, 'user': DB_USER, 'password': DB_PASSWORD, } app = { 'root': 'bll.api.controllers.root.RootController', 'modules': ['bll'], 'hooks': [RestHook()], 'debug': True } # Log everything to the console logging = { 'version': 1, 'root': {'level': LOG_LEVEL, 'handlers': ['console']}, 'loggers': { 'bll': {'level': LOG_LEVEL, 'handlers': ['consolewithtxn'], 'propagate': False}, # 'py.warnings': {'handlers': ['console']}, # '__force_dict__': True }, 'handlers': { 'console': { 'level': 'DEBUG', 'class': 'logging.StreamHandler', 'formatter': 'simple' }, 'consolewithtxn': { 'level': 'DEBUG', 'class': 'logging.StreamHandler', 'formatter': 'withtxn' }, }, 'formatters': { 'simple': { 'format': '%(asctime)s %(levelname)-5.5s [%(name)-30.30s]' '[%(threadName)-19.19s] %(message).512s' }, 'withtxn': { 'format': '%(asctime)s %(levelname)-5.5s [%(name)-30.30s]' '[%(threadName)-19.19s] [%(txn_id)-8.8s] %(message).512s' }, } } # BLL reads the keystone service catalog to determine which URL to use to reach # a given service. For production, this is normally the 'internalURL'. For # development, it is convenient to use the 'publicURL' so that a BLL running on # a developer system can reach services running on a remote deployment. # By default, 'internalURL' is used, unless the following block is set. services = { 'endpoint_type': 'publicURL', 'interface': 'public' } 070701000A47CF000041ED0000000000000000000000025B64B4DF00000000000000FD0000000200000000000000000000004200000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/tests/plugins070701000A47D5000081A40000000000000000000000015B64B4DF00000000000000FD0000000200000000000000000000004E00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/tests/plugins/__init__.py070701000A47D4000081A40000000000000000000000015B64B4DF00001C3D000000FD0000000200000000000000000000005900000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/tests/plugins/test_ardana_service.py# (c) Copyright 2016-2017 Hewlett Packard Enterprise Development LP # (c) Copyright 2017-2018 SUSE LLC from mock import mock, patch from tests.util import functional, get_token_from_env, TestCase, randomurl, \ get_mock_token, randomdict from bll.common.exception import InvalidBllRequestException from bll.api.request import BllRequest from bll import api from bll.plugins.ardana_service import ArdSvc from bll.common.job_status import get_job_status @functional('keystone,ardana') class ArdanaSvcTest(TestCase): def setUp(self): self.user = None self.token = get_token_from_env() def ardana_operation(self, operation=None, path=None, request_data=None, location=None): data = {} if path: data[api.PATH] = path if operation: data[api.OPERATION] = operation if request_data: data[api.REQUEST_DATA] = request_data if location: data['location'] = location request = { api.ACTION: 'GET', api.TARGET: 'ardana', api.AUTH_TOKEN: self.token, api.DATA: data } svc = ArdSvc(bll_request=BllRequest(request)) return svc.handle() def test_delete_compute_host_no_input(self): with self.assertRaises(Exception): self.ardana_operation(operation='delete_compute_host') class ArdSvcUnitTest(TestCase): mock_net_data = { 'host1': { 'net_data': { 'BOND0': { 'network1': { 'addr': '192.168.1.123', 'some': 'data1' }, 'network2': { 'endpoints': { 'some': 'moredata' }, 'blah': 'blah' } }, 'eth0': { 'network3': { 'addr': '192.168.3.123', 'some': 'data3' } } } }, 'host2': { 'net_data': { 'BOND0': { 'network3': { 'addr': '192.168.3.231', 'some': 'data3' } }, 'eth0': { 'network1': { 'addr': '192.168.3.123', 'some': 'data1' } } } }, } @mock.patch('bll.plugins.service.TokenHelpers') def test_query_parameters(self, _token_helper): base_url = randomurl() data = randomdict() path = "my/path" # The token_helper constructor returns an object with # a get_service_endpoint function that we want to override _token_helper.return_value.get_service_endpoint.return_value = base_url svc = ArdSvc(BllRequest(operation="do_path_operation", auth_token=get_mock_token(), action="GET", data={ 'path': path, 'request_data': data, 'request_parameters': ['key=value'] })) svc._request = mock.Mock() svc.handle() # Verify that the proper values are going to be passed to the requests # library svc._request.assert_called_once_with(path, {"key": "value"}, data, action='GET') @patch('bll.plugins.service.TokenHelpers.get_service_endpoint', return_value=randomurl()) def test_no_playbook(self, *_): svc = ArdSvc(BllRequest(operation='run_playbook', auth_token=get_mock_token(), action='POST', data={ })) with self.assertRaises(InvalidBllRequestException): svc.handle() def mock_request(*args, **kwargs): class MockResponse: def __init__(self, status_code, data): self.status_code = status_code self.data = data def json(self): return self.data action = args[0] url = args[1] if action == 'POST' and url.endswith('/playbooks/some_playbook'): return MockResponse(200, { 'id': 'ref_1', 'alive': True }) elif action == 'GET': # we're now in the non-validate section of handling an is_long # function so just pretend we're done if url.endswith('model/cp_output/server_info.yml'): return MockResponse(200, ArdSvcUnitTest.mock_net_data) return MockResponse(200, { 'id': 'ref_1', 'alive': False, # playbook completed 'code': 0 # playbook return code }) return None @patch('bll.plugins.service.TokenHelpers.get_service_endpoint', return_value=randomurl()) @patch('bll.plugins.ardana_service.requests.request', side_effect=mock_request) def test_playbook_cycle(self, *_): svc = ArdSvc(BllRequest(operation='run_playbook', auth_token=get_mock_token(), action='POST', data={ 'playbook_name': 'some_playbook', })) # Kick off the playbook resp = svc.handle() txn_id = resp[api.TXN_ID] # We now should be busy running the playbook status = get_job_status(txn_id) self.assertTrue(status['status'], api.STATUS_INPROGRESS) svc.update_job_status() self.assertTrue(status['status'], api.STATUS_INPROGRESS) # Now pretend we are done svc.update_job_status('done', percentage_complete=100, txn_id=txn_id) svc.sc_complete() self.assertTrue(status['status'], api.COMPLETE) self.assertFalse(status[api.DATA]['alive']) self.assertEquals(status[api.DATA]['code'], 0) @patch('bll.plugins.service.TokenHelpers.get_service_endpoint', return_value=randomurl()) @patch('bll.plugins.ardana_service.requests.request', side_effect=mock_request) def test_get_network_data(self, *_): svc = ArdSvc(BllRequest(operation='get_network_data', auth_token=get_mock_token())) networks = svc.handle()['data'] # should be 3 networks here self.assertEqual(len(networks), 3) for network in networks: self.assertIsNone(network.get('addr', None)) self.assertIsNone(network.get('endpoints', None)) 070701000A47D3000081A40000000000000000000000015B64B4DF00002F79000000FD0000000200000000000000000000005A00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/tests/plugins/test_catalog_service.py# (c) Copyright 2016-2017 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC import mock from tests.util import TestCase from bll import api from bll.plugins.catalog_service import CatalogSvc from bll.api.auth_token import TokenHelpers from bll.api.request import BllRequest from tests.util import functional, get_token_from_env from tests.util import get_mock_token class TestCatalogSvc(TestCase): def setUp(self): self.mock_serv_comp = { '__force_dict__': True, "monasca": { "components": { "monasca-transform": { "control_planes": { "ccp": { "clusters": { "cluster1": [ "standard-ccp-c1-m1-mgmt", "standard-ccp-c1-m2-mgmt", "standard-ccp-c1-m3-mgmt" ] }, "regions": [ "region1" ] } } } } }, "nova": { "components": { "nova-compute": { "control_planes": { "ccp": { "regions": [ "region1" ], "resources": { "compute": [ "standard-ccp-comp0001-mgmt", "standard-ccp-comp0002-mgmt", "standard-ccp-comp0003-mgmt" ] } } } } } }, "swift": { "components": { "swift-account": { "control_planes": { "ccp": { "clusters": { "cluster1": [ "standard-ccp-c1-m1-mgmt", "standard-ccp-c1-m2-mgmt", "standard-ccp-c1-m3-mgmt" ] }, "regions": [ "region1" ] } } }, "swift-container": { "control_planes": { "ccp": { "clusters": { "cluster1": [ "standard-ccp-c1-m1-mgmt", "standard-ccp-c1-m2-mgmt", "standard-ccp-c1-m3-mgmt" ] }, "regions": [ "region1" ] } } }, "swift-object": { "control_planes": { "ccp": { "clusters": { "cluster1": [ "standard-ccp-c1-m1-mgmt", "standard-ccp-c1-m3-mgmt" ] }, "regions": [ "region1" ] } } }, "swift-proxy": { "control_planes": { "ccp": { "clusters": { "cluster1": [ "standard-ccp-c1-m1-mgmt", "standard-ccp-c1-m2-mgmt", "standard-ccp-c1-m3-mgmt", "some_host" ] }, "regions": [ "region1" ] } } } } } } # pretend comp0003 is a baremetal node, thus not in hypervisor-list self.mock_hyp_list = [ { 'name': 'standard-ccp-comp0001-mgmt', 'region': 'region1', 'service_host': 'standard-ccp-comp0001-mgmt' }, { 'name': 'standard-ccp-comp0002-mgmt', 'region': 'region1', 'service_host': 'standard-ccp-comp0002-mgmt' }, ] def mock_call_service(self, target=None, operation=None, data=None, action=None, include_status=False): if target == 'ardana': return {'services': self.mock_serv_comp} elif target == 'nova' and operation == 'hypervisor-list': return self.mock_hyp_list @mock.patch.object(CatalogSvc, '_get_services', return_value=['identity', 'monitoring']) def test_get_plugins(self, _): request = { 'target': 'catalog', 'data': {'operation': 'get_plugins'} } catalog_service = CatalogSvc(bll_request=BllRequest(request)) reply = catalog_service.handle() self.assertEqual('complete', reply[api.STATUS]) plugins = reply[api.DATA] self.assertIn('general', plugins) self.assertNotIn('unavailable', plugins) # This is does not really add any test to the genuine unit test above, # but it can be useful for printing what the real plugin list is @functional('keystone') def test_get_real_plugins(self): token = get_token_from_env() request = { 'target': 'catalog', 'data': {'operation': 'get_plugins'}, 'auth_token': token } catalog_service = CatalogSvc(bll_request=BllRequest(request)) reply = catalog_service.handle() self.assertEqual('complete', reply[api.STATUS]) plugins = reply[api.DATA] # print plugins self.assertIn('general', plugins) self.assertNotIn('unavailable', plugins) @functional('keystone') def test_get_services(self): token = get_token_from_env() request = { 'target': 'catalog', 'data': {'operation': 'get_services'}, 'auth_token': token } catalog_service = CatalogSvc(bll_request=BllRequest(request)) reply = catalog_service.handle() self.assertEqual('complete', reply[api.STATUS]) services = reply[api.DATA] self.assertGreater(len(services), 2) self.assertIn('identity', services) self.assertIn('keystone', services) self.assertIn('monitoring', services) @mock.patch('bll.plugins.service.SvcBase.call_service') @mock.patch('bll.plugins.catalog_service.get_conf') def test_get_compute_clusters_from_conf(self, mock_conf, mock_legacy): mock_conf.return_value = self.mock_serv_comp['nova']['components'] mock_legacy.side_effect = self.mock_call_service self._test_get_compute_clusters() @mock.patch('bll.plugins.service.SvcBase.call_service') @mock.patch('bll.plugins.catalog_service.get_conf') def test_get_compute_clusters_from_ardana(self, mock_conf, mock_legacy): mock_conf.return_value = None mock_legacy.side_effect = self.mock_call_service self._test_get_compute_clusters() def _test_get_compute_clusters(self): request = { 'target': 'catalog', 'data': {'operation': 'get_compute_clusters'}, 'auth_token': get_mock_token() } svc = CatalogSvc(bll_request=BllRequest(request)) data = svc.handle()[api.DATA] self.assertTrue('ccp:compute' in data) self.assertEqual(len(data['ccp:compute']), 2) self.assertTrue('standard-ccp-comp0001-mgmt' in data['ccp:compute']) self.assertTrue('standard-ccp-comp0003-mgmt' not in data['ccp:compute']) @mock.patch('bll.plugins.catalog_service.get_conf') def test_get_swift_clusters_from_conf(self, mock_conf): mock_conf.return_value = self.mock_serv_comp['swift']['components'] self._test_get_swift_clusters() @mock.patch('bll.plugins.service.SvcBase.call_service') @mock.patch('bll.plugins.catalog_service.get_conf') def test_get_swift_clusters_from_ardana(self, mock_conf, mock_legacy): mock_conf.return_value = None mock_legacy.side_effect = self.mock_call_service self._test_get_swift_clusters() def _test_get_swift_clusters(self): request = { 'target': 'catalog', 'data': {'operation': 'get_swift_clusters'}, 'auth_token': get_mock_token() } svc = CatalogSvc(bll_request=BllRequest(request)) data = svc.handle()[api.DATA] self.assertTrue('ccp:cluster1' in data) self.assertEqual(len(data['ccp:cluster1']), 4) self.assertTrue('standard-ccp-c1-m1-mgmt' in data['ccp:cluster1']) self.assertTrue('some_host' in data['ccp:cluster1']) @mock.patch('bll.plugins.catalog_service.get_conf') @mock.patch('bll.plugins.catalog_service.CatalogSvc._get_services') def test_monasca_transform_avail(self, mock_serv, mock_conf): mon_comps = self.mock_serv_comp['monasca']['components'] mock_conf.return_value = mon_comps mock_serv.return_value = [] request = { 'target': 'catalog', 'data': {'operation': 'get_plugins'}, 'auth_token': get_mock_token() } svc = CatalogSvc(bll_request=BllRequest(request)) data = svc.handle()[api.DATA] self.assertIn('monasca-transform', data) @mock.patch('bll.plugins.catalog_service.get_conf') @mock.patch('bll.plugins.catalog_service.CatalogSvc._get_services') def test_monasca_transform_notavail(self, mock_serv, mock_conf): mon_comps = None mock_conf.return_value = mon_comps mock_serv.return_value = [] request = { 'target': 'catalog', 'data': {'operation': 'get_plugins'}, 'auth_token': get_mock_token() } svc = CatalogSvc(bll_request=BllRequest(request)) data = svc.handle()[api.DATA] self.assertNotIn('monasca-transform', data) @functional('keystone') def test_get_regions(self): token = get_token_from_env() request = { 'target': 'catalog', 'data': {'operation': 'get_regions'}, 'auth_token': token } catalog_service = CatalogSvc(bll_request=BllRequest(request)) reply = catalog_service.handle() self.assertEqual('complete', reply[api.STATUS]) regions = reply[api.DATA] self.assertGreater(len(regions), 0) self.assertEqual('region1', regions[0]['id']) @mock.patch.object(TokenHelpers, 'get_service_endpoint') def test_get_enterprise_app_endpoints(self, mock_get_service_endpoint): svc = CatalogSvc(BllRequest(operation='get_enterprise_app_endpoints', auth_token=get_mock_token())) output = svc.handle() self.assertGreater(len(output['data']), 0) 070701000A47DD000081A40000000000000000000000015B64B4DF0000093F000000FD0000000200000000000000000000005900000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/tests/plugins/test_cinder_service.py# (c) Copyright 2016-2017 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC from tests.util import functional, get_token_from_env, TestCase, \ randomidentifier from bll.api.request import BllRequest from bll import api from bll.plugins.cinder_service import CinderSvc @functional('keystone,cinder') class TestCinderSvc(TestCase): def setUp(self): self.token = get_token_from_env() def test_volume_crud(self): volume_type = "test_" + randomidentifier() # Add the volume type and grab its id svc = CinderSvc(BllRequest(target="cinder", operation="volume_type_add", auth_token=self.token, volume_type=volume_type)) reply = svc.handle() self.assertEqual(api.COMPLETE, reply[api.STATUS]) svc = CinderSvc(BllRequest(target="cinder", auth_token=self.token, operation="volume_type_list")) reply = svc.handle() volume_types = reply[api.DATA] for id, vol_type in volume_types.iteritems(): if vol_type == volume_type: new_id = id break else: self.fail("Newly created volume does not appear in list") svc = CinderSvc(BllRequest(target="cinder", operation="map_volume_backend", auth_token=self.token, volume_type_id=new_id, backend_name=randomidentifier())) reply = svc.handle() self.assertEqual(api.COMPLETE, reply[api.STATUS]) # Now delete it and verify it is gone svc = CinderSvc(BllRequest(target="cinder", operation="volume_type_delete", auth_token=self.token, volume_type_id=new_id)) reply = svc.handle() self.assertEqual(api.COMPLETE, reply[api.STATUS]) svc = CinderSvc(BllRequest(target="cinder", operation="volume_type_list", auth_token=self.token)) reply = svc.handle() self.assertNotIn(new_id, reply[api.DATA]) 070701000A47DA000081A40000000000000000000000015B64B4DF00002939000000FD0000000200000000000000000000005A00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/tests/plugins/test_compute_service.py# Copyright 2016-2017 Hewlett Packard Enterprise Development LP # (c) Copyright 2017-2018 SUSE LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. from bll import api from bll.api.auth_token import TokenHelpers from bll.api.request import BllRequest from bll.plugins import compute_service from mock import patch from random import random from tests.util import TestCase, get_mock_token, randomurl import copy import mock def call_service_test_utilization(target=None, operation=None, data=None): if not operation: operation = data['operation'] hosts = ['host1', 'host2', 'host3'] if target == 'catalog' and operation == 'get_compute_clusters': return { 'ccp:compute': hosts } elif target == 'monitor' and operation == 'measurement_list': metric_name = data['name'] meas_list = [] if '_perc' in metric_name: for host in hosts: meas_list.append( get_meas(host, float('%.1f' % (random() * 100)))) elif '_mb' in metric_name: if 'total' in metric_name: for host in hosts: meas_list.append(get_meas(host, float('%.1f' % 8192))) else: for host in hosts: meas = get_meas(host, float('%.1f' % (random() * 8192))) meas_list.append(meas) return meas_list elif target == 'nova' and operation == 'hypervisor-list': return [ { 'hypervisor_id': 1, 'name': 'host1', 'hypervisor_hostname': 'host1', 'service_host': 'host1', }, { 'hypervisor_id': 2, 'name': 'host2', 'hypervisor_hostname': 'host2', 'service_host': 'host2', }, { 'hypervisor_id': 3, 'name': 'host3', 'hypervisor_hostname': 'host3', 'service_host': 'host3', } ] def call_service_test_details(target=None, data=None, operation=None, path=None): hostname = 'some_name' cp_host_name = 'test_cp_host' if target == 'nova' and operation == 'hypervisor-list': return [ { 'hypervisor_id': 1, 'instances': 2, 'name': hostname } ] elif target == 'monitor' and data['operation'] == 'measurement_list': return [ { 'columns': ['timestamp', 'value', 'value_meta'], 'measurements': [['some_data_string', 123]] } ] elif target == 'ardana': if path == '/model/cp_output/server_info.yml': return { cp_host_name: { 'hostname': hostname } } elif path == '/model/entities/servers/' + cp_host_name: return { 'id': cp_host_name, 'role': 'some_role', 'server-group': 'some_group' } def get_meas(host, value): meas_template = { 'columns': ['timestamp', 'value', 'value_meta'], 'measurements': [['some_time', 'some_value', {}]], 'dimensions': {'hostname': 'some_host'} } value_idx = meas_template['columns'].index('value') meas = copy.deepcopy(meas_template) meas['measurements'][-1][value_idx] = value meas['dimensions']['hostname'] = host return meas def call_service(target=None, operation=None, region=None, data=None): if target == 'nova' and operation == 'hypervisor-allocation-stats': return { 1: { "allocated_cpu": 0, "allocated_memory": 2816, "allocated_storage": 10, "instances": 0, "total_cpu": 32.0, "total_memory": 252198.0, "total_storage": 1110 } } elif target == 'nova' and operation == 'hypervisor-list': return [ { "allocated_cpu": 0, "allocated_memory": 2816, "allocated_storage": 10, "hypervisor_id": 1, "instances": 0, "hypervisor_hostname": "domain-c84.68bb08ca-07ed-4b9d-835f-1815a5f15a85", "ping_status": "up", "service_host": "some_host", "region": "my_region", "state": "up", "status": "enabled", "total_cpu": 32, "total_memory": 252198, "total_storage": 1110, "type": "VMware vCenter Server" } ] elif target == 'eon' and operation == 'resource_list': return [ { "hypervisor_id": "1", "instances": 0, "hypervisor_hostname": "domain-c84.68bb08ca-07ed-4b9d-835f-1815a5f15a85", "ping_status": "up", "region": "my_region", "service_host": "some_host", "state": "up", "status": "enabled", "type": "VMware vCenter Server" } ] class TestComputeSvc(TestCase): @patch.object(TokenHelpers, 'get_service_endpoint', return_value=randomurl()) @patch('bll.plugins.service.SvcBase.call_service', side_effect=call_service) def test_get(self, *_): svc = compute_service.ComputeSvc(BllRequest( action='GET', operation='get_compute_list', auth_token=get_mock_token())) reply = svc.handle() self.assertEqual(api.COMPLETE, reply[api.STATUS]) self.assertEqual(1, len(reply['data'])) def test_filter_out_hosts(self): compute_list = [{ 'name': 'hostkvm', 'type': 'kvm' }, { 'name': 'hostironic', 'type': 'ironic' }, { 'name': 'hostqemu', 'type': 'qemu' }] # filter out hosts with type set to ironic hosts = compute_service.ComputeSvc._filter_out_hosts(compute_list, 'type', 'ironic') self.assertEqual(2, len(hosts)) for host in hosts: self.assertNotEqual(host['type'], 'ironic') @patch('bll.api.auth_token.TokenHelpers.get_service_endpoint', return_value=randomurl()) def test_get_compute_details_legacy(self, *_): request = { api.TARGET: 'compute', api.ACTION: 'GET', api.AUTH_TOKEN: get_mock_token(), api.DATA: { api.OPERATION: 'get_compute_list', api.VERSION: 'v1', api.DATA: { "id": "e90f7f6f-c75f-4830-8f47-e0af2851b132", "hypervisor": "esxcluster", "hypervisor_id": "1" } } } svc = compute_service.ComputeSvc( bll_request=BllRequest(request)) svc.call_service = mock.Mock(return_value=[]) reply = svc.handle() self.assertEqual(api.COMPLETE, reply[api.STATUS]) @patch.object(TokenHelpers, 'get_service_endpoint', return_value=None) def test_get_compute_details(self, a): request = { api.TARGET: 'compute', api.ACTION: 'GET', api.AUTH_TOKEN: get_mock_token(), api.DATA: { api.OPERATION: 'get_compute_list', api.VERSION: 'v1', api.DATA: { "id": "e90f7f6f-c75f-4830-8f47-e0af2851b132", "hypervisor": "esxcluster", "hypervisor_id": "1" } } } svc = compute_service.ComputeSvc(bll_request=BllRequest(request)) svc.call_service = mock.Mock(return_value=[]) reply = svc.handle() self.assertEqual(api.COMPLETE, reply[api.STATUS]) @patch.object(TokenHelpers, 'get_service_endpoint', return_value=None) @patch('bll.plugins.service.SvcBase.call_service', side_effect=call_service_test_utilization) def test_get_cluster_utilization(self, *_): request = { api.TARGET: 'compute', api.ACTION: 'GET', api.AUTH_TOKEN: get_mock_token(), api.DATA: { api.OPERATION: 'get_cluster_utilization' } } svc = compute_service.ComputeSvc(bll_request=BllRequest(request)) reply = svc.handle() self.assertEqual(api.COMPLETE, reply[api.STATUS]) data = reply[api.DATA] self.assertTrue('ccp:compute' in data) # 3 hosts self.assertEqual(len(data['ccp:compute']), 3) self.assertTrue('host1' in data['ccp:compute']) self.assertTrue('used_memory_perc' in data['ccp:compute']['host1']) self.assertTrue('used_storage_perc' in data['ccp:compute']['host1']) self.assertTrue('used_cpu_perc' in data['ccp:compute']['host1']) @patch('bll.api.auth_token.TokenHelpers.get_service_endpoint', return_value=True) @patch('bll.plugins.service.SvcBase.call_service', side_effect=call_service_test_details) def test_get_non_eon_details(self, *_): svc = compute_service.ComputeSvc(BllRequest( auth_token=get_mock_token(), operation='details', region='region1', data={ 'data': { 'id': '1', 'type': 'kvm' } })) reply = svc.handle() self.assertEqual(reply['data']['ardana']['server-group'], 'some_group') self.assertEqual(reply['data']['monasca']['used_cpu_perc'], 123) self.assertEqual(reply['data']['instances'], 2) self.assertEqual(reply[api.STATUS], api.COMPLETE) 070701000A47D7000081A40000000000000000000000015B64B4DF000019D5000000FD0000000200000000000000000000005900000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/tests/plugins/test_expose_service.py# (c) Copyright 2015-2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC from bll.plugins.service import SvcBase from tests.util import TestCase from bll.api.request import BllRequest from bll.common.job_status import get_job_status from bll import api from tests import util import time import logging class TestExposeSvc(TestCase): def testNull(self): request = { api.TARGET: 'general', api.DATA: { api.OPERATION: 'null' } } reply = SvcBase.spawn_service(BllRequest(request)) self.assertEqual(reply[api.STATUS], api.COMPLETE) def testOp(self): request = { api.TARGET: 'expose', api.DATA: { api.OPERATION: 'valid_op' } } reply = SvcBase.spawn_service(BllRequest(request)) self.assertEqual(reply[api.STATUS], api.COMPLETE) self.assertEqual(reply[api.DATA], 'ok') def testOpGet(self): request = { api.TARGET: 'expose', api.ACTION: 'GET', api.DATA: { api.OPERATION: 'valid_op' } } reply = SvcBase.spawn_service(BllRequest(request)) self.assertEqual(reply[api.STATUS], api.COMPLETE) self.assertEqual(reply[api.DATA], 'ok') def testOpPost(self): request = { api.TARGET: 'expose', api.ACTION: 'POST', api.DATA: { api.OPERATION: 'valid_op' } } reply = SvcBase.spawn_service(BllRequest(request)) self.assertEqual(reply[api.STATUS], api.COMPLETE) self.assertEqual(reply[api.DATA], 'ok') def testAction(self): request = { api.TARGET: 'expose', api.ACTION: 'act', } reply = SvcBase.spawn_service(BllRequest(request)) self.assertEqual(reply[api.STATUS], api.COMPLETE) self.assertEqual(reply[api.DATA], 'action') def testInvalidOp(self): request = { api.TARGET: 'expose', api.DATA: { api.OPERATION: 'not_exposed' } } # Suppress exception log with util.log_level(logging.CRITICAL, 'bll'): reply = SvcBase.spawn_service(BllRequest(request)) self.assertEqual(reply[api.STATUS], api.STATUS_ERROR) self.assertIn('Unsupported', reply[api.DATA][0][api.DATA]) def testMissingOp(self): request = { api.TARGET: 'expose', api.DATA: { api.OPERATION: 'foo' } } # Suppress exception log with util.log_level(logging.CRITICAL, 'bll'): reply = SvcBase.spawn_service(BllRequest(request)) self.assertEqual(reply[api.STATUS], api.STATUS_ERROR) self.assertIn('Unsupported', reply[api.DATA][0][api.DATA]) def testSlowOp(self): request = { api.TARGET: 'expose', api.DATA: { api.OPERATION: 'slow_op' } } # Suppress exception log txn_id = None with util.log_level(logging.CRITICAL, 'bll'): reply = SvcBase.spawn_service(BllRequest(request)) txn_id = reply.get(api.TXN_ID) self.assertEqual(reply[api.PROGRESS]['percentComplete'], 0) self.assertEqual(api.STATUS_INPROGRESS, reply[api.STATUS]) time.sleep(0.1) reply = get_job_status(txn_id) self.assertEqual(reply[api.PROGRESS]['percentComplete'], 100) def testMultiDecorator(self): request = { api.TARGET: 'expose', api.DATA: { api.OPERATION: 'op1' } } reply = SvcBase.spawn_service(BllRequest(request)) self.assertEqual(reply[api.STATUS], api.COMPLETE) self.assertEqual(reply[api.DATA], 'multi') request[api.DATA][api.OPERATION] = 'op2' reply = SvcBase.spawn_service(BllRequest(request)) self.assertEqual(reply[api.STATUS], api.COMPLETE) self.assertEqual(reply[api.DATA], 'multi') def test_spawn_long(self): """ This function is nearly identical to test_spawn_long in test_service_collection.py, with the only difference being the target and operation in the request. This illustrates that the behavior of the new @expose method is the same as the old handle/complete for asynchronous calls. """ pauses = 5 pause_sec = 0.1 bll_request = { api.TARGET: 'expose', api.DATA: { api.OPERATION: 'progress', 'pause_sec': pause_sec, 'num_pauses': pauses, } } reply = SvcBase.spawn_service(BllRequest(bll_request)) txn_id = reply.get(api.TXN_ID) # test the incomplete reply self.assertIn(api.TXN_ID, reply) self.assertIn(api.PROGRESS, reply) self.assertIn(api.POLLING_INTERVAL, reply) self.assertIn(api.STARTTIME, reply) self.assertIn(api.STATUS, reply) self.assertEqual(api.STATUS_INPROGRESS, reply[api.STATUS]) # wait a little extra before the first poll time.sleep(0.1 + pause_sec) # read the job status update reply = get_job_status(txn_id) self.assertEqual(reply[api.TXN_ID], txn_id) self.assertIn(api.STARTTIME, reply) self.assertIn(api.STATUS, reply) self.assertEqual(api.STATUS_INPROGRESS, reply[api.STATUS]) percent = reply[api.PROGRESS][api.PERCENT_COMPLETE] self.assertGreater(percent, 0) self.assertLess(percent, 100) while reply.get(api.STATUS) != api.COMPLETE: time.sleep(pause_sec) reply = get_job_status(txn_id) self.assertEqual(reply[api.TXN_ID], txn_id) self.assertIn(api.DURATION, reply) self.assertIn(api.ENDTIME, reply) self.assertIn(api.STARTTIME, reply) self.assertIn(api.STATUS, reply) self.assertEqual(reply[api.STATUS], api.COMPLETE) self.assertEqual(reply[api.PROGRESS][api.PERCENT_COMPLETE], 100) def test_returned_data(self): request = { api.TARGET: 'expose', api.DATA: { api.OPERATION: 'data_in_response' } } reply = SvcBase.spawn_service(BllRequest(request)) # even though the data_in_response method doesn't return anything, # we should still see data self.assertEqual(reply[api.DATA], 'blah') 070701000A47D6000081A40000000000000000000000015B64B4DF00000E8C000000FD0000000200000000000000000000005900000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/tests/plugins/test_ironic_service.py# (c) Copyright 2016-2017 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC from bll.api.auth_token import TokenHelpers from bll.api.request import BllRequest from bll import api from bll.plugins.ironic_service import IronicSvc from tests.util import TestCase, randomword, randomhex, get_mock_token, \ randomurl from ironicclient.v1.node import NodeManager import mock class TestIronicSvc(TestCase): @mock.patch('bll.plugins.service.SvcBase.call_service', return_value={ 'instances': [{ 'id': 'instance_1', 'status': 'ACTIVE' }, { 'id': 'instance_2', 'status': 'ACTIVE' }] }) @mock.patch.object(TokenHelpers, 'get_endpoints') @mock.patch.object(IronicSvc, '_get_ironic_client') def test_baremetal_list(self, _mock_get_func, _mock_endpoints, call_service): # this will also indirectly test node.list (generic_list) mock_client = mock.MagicMock() mock_client.node = mock.create_autospec(NodeManager) _mock_endpoints.return_value = [{'region': randomword(), 'url': randomurl()}] the_list = [ self.MockResource( { 'instance_uuid': 'instance_1', 'uuid': randomhex(), 'name': randomword(), 'power_state': 'power on' }), self.MockResource( { 'instance_uuid': 'instance_2', 'uuid': randomhex(), 'name': randomword(), 'power_state': 'power on' })] mock_client.node.list.return_value = the_list _mock_get_func.return_value = mock_client svc = IronicSvc(BllRequest(operation='baremetal-list', auth_token=get_mock_token())) data = svc.handle()[api.DATA] self.assertEqual(len(data), 2) for inst in data: self.assertTrue('baremetal', 'compute' in inst.keys()) self.assertTrue(inst['baremetal']['instance_uuid'] == inst['compute']['id']) @mock.patch.object(IronicSvc, '_get_ironic_client') @mock.patch.object(TokenHelpers, 'get_endpoints') def test_generic_get(self, _mock_endpoints, _mock_get_func): mock_client = mock.MagicMock() mock_client.node = mock.create_autospec(NodeManager) _mock_endpoints.return_value = [{'region': randomword(), 'url': randomurl()}] nodeid = randomhex() res = self.MockResource( { 'instance_uuid': randomhex(), 'uuid': nodeid, 'driver': 'agent_ilo', 'name': randomword(), 'power_state': 'power on', 'provision_state': 'active', }) mock_client.node.get.return_value = res _mock_get_func.return_value = mock_client svc = IronicSvc(BllRequest(operation='node.get', auth_token=get_mock_token(), data={'node_id': randomhex})) data = svc.handle()[api.DATA] self.assertIsInstance(data, dict) self.assertEqual(data['driver'], 'agent_ilo') self.assertEqual(data['uuid'], nodeid) class MockResource(): data = {} def __init__(self, data): self.data = data def to_dict(self): return self.data 070701000A47DB000081A40000000000000000000000015B64B4DF00002F03000000FD0000000200000000000000000000005A00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/tests/plugins/test_monitor_service.py# (c) Copyright 2015-2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC from datetime import timedelta, datetime import unittest from mock import patch, MagicMock from monascaclient import client as msclient from tests.util import functional, get_token_from_env, TestCase, \ create_user, delete_user from bll.api.request import BllRequest, InvalidBllRequestException from bll import api from bll.plugins.monitor_service import TYPE_UNKNOWN, TYPE_UP, \ MonitorSvc @functional('keystone,monasca') class TestMonitorSvc(TestCase): def setUp(self): self.user = None self.alarm_definition_id = None self.notification_id = None self.token = get_token_from_env() self.mock_moncli = MagicMock(spec=msclient.Client) def tearDown(self): if self.alarm_definition_id: self.monitor_operation('alarm_definition_delete', dict(id=self.alarm_definition_id)) if self.notification_id: self.monitor_operation('notification_delete', dict(id=self.notification_id)) if self.user: delete_user(self.user) def monitor_operation(self, operation, data=None): """ Call the monitor service to perform the given operation, without any assertions. The lack of assertions is useful in scenarios where success it not expected and/or required """ request = { api.TARGET: 'monitor', api.AUTH_TOKEN: self.token, api.DATA: { api.OPERATION: operation, api.VERSION: 'v1', } } if data: request[api.DATA].update(data) svc = MonitorSvc(bll_request=BllRequest(request)) return svc.handle() def monitor(self, operation, data=None): """ Performs a monitor operation, and asserts for success """ reply = self.monitor_operation(operation, data) self.assertEqual(api.COMPLETE, reply[api.STATUS]) self.assertIn(api.PROGRESS, reply) self.assertIn(api.STARTTIME, reply) return reply[api.DATA] def test_alarm_count_all(self): output = self.monitor('alarm_count') self.assertIsNotNone(output) self.assertIn('counts', output) self.assertIn('columns', output) total_alarms = output['counts'][0][0] # There is almost always some sort of alarm self.assertNotEqual(total_alarms, 0) parms = {'group_by': "severity,state"} output = self.monitor('alarm_count', parms) # Flatten the array of arrays and look for any valid vals = [val for innerlist in output['counts'] for val in innerlist] # There should be a valid severity in there somewhere self.assertTrue(any(val in vals for val in ['LOW', 'MEDIUM', 'HIGH'])) # There should be a valid state in there somewhere self.assertTrue(any(val in vals for val in ['UNDETERMINED', 'OK', 'ALARM'])) def test_alarm_count_filter(self): # We want only counts where dimension_name = service or hostname parms = { 'dimension_name_filter': 'service,hostname', 'group_by': 'dimension_name,dimension_value,severity,state' } output = self.monitor('alarm_count', parms) dim_idx = output['columns'].index('dimension_name') for count in output['counts']: self.assertTrue(count[dim_idx] in ('service', 'hostname')) # But what happens if we specify the dimension_name_filter without # specifying dimension_name in the group_by parms = { 'dimension_name_filter': 'service,hostname', 'group_by': 'severity,state' } with self.assertRaises(InvalidBllRequestException): self.monitor('alarm_count', parms) def test_alarm_definitions(self): name = "TEST alarm -- delete me" definition_data = { "name": name, "description": "TEST -- delete me", "expression": "disk.space_used_perc>90", "match_by": ["hostname"], "severity": "LOW" } data = self.monitor('alarm_definition_create', definition_data) self.assertIn('id', data) id = data['id'] # Enable tearDown to cleanup in case of subsequent failures self.alarm_definition_id = id data = self.monitor('alarm_definition_list') self.assertIn(id, [dfn['id'] for dfn in data]) data = self.monitor('alarm_definition_show', dict(id=id)) self.assertEquals(id, data.get('id')) self.assertEquals(name, data.get('name')) desc = 'Updated description' data['description'] = desc data = self.monitor('alarm_definition_update', data) self.assertEquals(desc, data.get('description')) desc = 'Patched description' data = self.monitor('alarm_definition_patch', dict(id=id, description=desc)) data = self.monitor('alarm_definition_show', dict(id=id)) self.assertEquals(id, data.get('id')) self.assertEquals(desc, data.get('description')) # clean up self.monitor('alarm_definition_delete', dict(id=id)) self.alarm_definition_id = None def test_notifications(self): name = "TEST -- delete" definition_data = { "name": name, "type": "EMAIL", "address": "foo@bar.com" } data = self.monitor('notification_create', definition_data) self.assertIn('id', data) id = data['id'] # Enable tearDown to cleanup in case of subsequent failures self.notification_id = id data = self.monitor('notification_list') self.assertIn(id, [dfn['id'] for dfn in data]) data = self.monitor('notification_show', dict(id=id)) self.assertEquals(id, data.get('id')) self.assertEquals(name, data.get('name')) address = 'updated@bar.com' data['address'] = address data = self.monitor('notification_update', data) self.assertEquals(address, data.get('address')) # clean up self.monitor('notification_delete', dict(id=id)) self.notification_id = None data = self.monitor('notificationtype_list') self.assertIn('EMAIL', [datum['type'] for datum in data]) self.assertIn('WEBHOOK', [datum['type'] for datum in data]) self.assertIn('PAGERDUTY', [datum['type'] for datum in data]) def test_metrics(self): # NOTE - since there is no API for deleting a metric, this test will # not exercise the metric_create API and pollute the system with test # data data = self.monitor('metric_list') self.assertGreater(len(data), 1) # Grab the first metric from the list name = data[0]['name'] dimensions = data[0]['dimensions'] end_time = datetime.utcnow() start_time = end_time + timedelta(-1) # Exercise metric_statistics data = self.monitor('metric_statistics', { 'name': name, 'statistics': 'avg', 'dimensions': dimensions, 'start_time': start_time.isoformat(), 'end_time': end_time.isoformat() }) self.assertGreater(len(data[0]['statistics']), 1) # Exercise measurement-list data = self.monitor('measurement_list', { 'name': name, 'dimensions': dimensions, 'start_time': start_time.isoformat(), 'end_time': end_time.isoformat() }) self.assertGreater(len(data[0]['measurements']), 1) @unittest.skip("skipping test_alarms() due to JAH-2347") def test_alarms(self): # NOTE - since there is no API for creating an alarm, this test will # not exercise the alarm_delete API data = self.monitor('alarm_list') self.assertGreater(len(data), 1) # Grab the first alarm from the list id = data[0]['id'] data = self.monitor('alarm_show', dict(id=id)) self.assertEquals(id, data.get('id')) old_lifecycle_state = data['lifecycle_state'] new_lifecycle_state = 'foo' data['lifecycle_state'] = new_lifecycle_state data = self.monitor('alarm_update', data) self.assertEquals(new_lifecycle_state, data.get('lifecycle_state')) # Now put it back where it started, using patch data = self.monitor('alarm_patch', dict(id=id, lifecycle_state=old_lifecycle_state)) data = self.monitor('alarm_show', dict(id=id)) self.assertEquals(old_lifecycle_state, data.get('lifecycle_state')) # Verify that alarm history succeeds self.monitor('alarm_history', dict(id=id)) # Verify that alarm history list succeeds end_time = datetime.utcnow() start_time = end_time + timedelta(-1) self.monitor('alarm_history_list', { 'start_time': start_time.isoformat(), 'end_time': end_time.isoformat() }) def test_monasca_user_role(self): # Test that a user with monasca-user role can retrieve monasca data (self.user, password) = create_user({'admin': 'monasca-user', 'demo': 'admin'}, {'Default': 'admin'}) data = self.monitor('notification_list') self.assertGreater(len(data), 0) def mock_cli_get_all_instances_status(start_time, group_by, dimensions, name): meas_groups = [] for i in range(3): host = {} host['dimensions'] = {} host['dimensions']['hostname'] = "host%s" % i host['dimensions']['resource_id'] = "host%s_id" % i host['measurements'] = [['time1', 0.0, {}]] meas_groups.append(host) return meas_groups @patch('monascaclient.v2_0.metrics.MetricsManager.list_measurements', side_effect=mock_cli_get_all_instances_status) def test_get_all_instances_status(self, mockname): # functional test data = self.monitor('get_all_instances_status') self.assertEqual(len(data), 3) # there are 3 hosts for i in range(len(data)): # for each host, verify good status self.assertEqual(data["host%d_id" % i]['vm.host_alive_status'], 0) self.assertEqual(data["host%d_id" % i]['vm.ping_status'], 0) self.assertEqual(data["host%d_id" % i] ['host_alive_status_value_meta_detail'], '') def test_get_appliances_status(self): # bad hosts data = self.monitor('get_appliances_status', data={ 'hostnames': ['moe', 'larry', 'curly'] }) for status in data.values(): self.assertEquals(status, TYPE_UNKNOWN) # see if we can come up with a list of good hosts metrics_list = self.monitor('metric_list', data={ 'name': 'host_alive_status', }) # this won't work if the whole cloud is in one host, but since we # never do that in our functional tests hostnames = [] for metric in metrics_list: hostname = metric['dimensions']['hostname'] if hostname not in hostnames: hostnames.append(hostname) # we expect all these hosts to be up. If not, our cloud is falling # apart and should be rebuilt. data = self.monitor('get_appliances_status', data={ 'hostnames': hostnames }) for status in data.values(): self.assertIn(status, (TYPE_UP, TYPE_UNKNOWN)) if status == TYPE_UNKNOWN: print "Perhaps it's time to rebuild the cloud??" 070701000A47DC000081A40000000000000000000000015B64B4DF0000578B000000FD0000000200000000000000000000005700000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/tests/plugins/test_nova_service.py# (c) Copyright 2015-2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC from datetime import timedelta, date from random import randint import random import logging import mock from mock import patch, MagicMock, Mock from bll import api from bll.api.request import BllRequest from bll.plugins.nova_service import NovaSvc, power_states from bll.plugins.user_group_service import UserGroupSvc from bll.plugins.monitor_service import MonitorSvc from bll.api.auth_token import TokenHelpers import novaclient.client as nclient from keystoneclient.v3 import client as ksclient from monascaclient import client as msclient from tests.util import TestCase, randomword, randomip, \ functional, get_token_from_env, log_level, get_mock_token class Flavor(): # Generate a mostly random flavor def __init__(self): self.id = randomword() self.name = randomword() self.vcpus = 2 ** randint(0, 3) self.ram = 1024 * randint(1, 16) self.disk = 1000 * randint(1, 8) class Server(): # Generate a mostly random server def __init__(self, name=None, image_id=None, power_state=None, flavor_id=None, metadata=None, tenant_id=None): self.name = name or randomword() self.id = randomword() self.metadata = metadata self.status = randomword() self._info = { 'OS-EXT-SRV-ATTR:host': randomword(), 'OS-EXT-STS:power_state': power_state, 'OS-EXT-AZ:availability_zone': randomword(), 'OS-EXT-STS:task_state': randomword()} self.image = {'id': image_id} self.addresses = randomword() self.created = None self.key_name = randomword() self.flavor = {'id': flavor_id} self.tenant_id = tenant_id class Hypervisor(): def __init__(self): self.vcpus_used = randint(0, 4) self.vcpus = randint(0, 4) self.memory_mb_used = randint(1024, 2048) self.memory_mb = randint(4096, 8192) self.local_gb_used = randint(1, 5) self.local_gb = randint(10, 20) self.running_vms = randint(0, 4) self.NAME_ATTR = 'hypervisor_hostname' self.hypervisor_hostname = 'testhost-%s' % randomword() self.host_ip = randomip() self.id = randint(1, 10000) self.status = 'enabled' self.state = 'up' self.hypervisor_type = 'QEMU' self.service = {'host': self.hypervisor_hostname} class Struct(object): """ Convert dictionary to object From http://stackoverflow.com/questions/1305532/convert-python-dict-to-object """ def __init__(self, data): for name, value in data.iteritems(): setattr(self, name, self._wrap(value)) def _wrap(self, value): if isinstance(value, (tuple, list, set, frozenset)): return type(value)([self._wrap(v) for v in value]) else: return Struct(value) if isinstance(value, dict) else value class TestNovaSvc(TestCase): def setUp(self): self.flavor_list = [] for x in range(5): self.flavor_list.append(Flavor()) self.image_list = [Struct({'id': '1', 'name': 'cirros'})] self.project_list = [ Struct({'id': '1', 'name': 'default_project'}), Struct({'id': '2', 'name': 'admin_project'}) ] self.hyp_list = [] for x in range(4): self.hyp_list.append(Hypervisor()) self.mock_novaclient = MagicMock(spec=nclient.Client) self.mock_novaclient.flavors = Mock(**{ 'list.return_value': self.flavor_list}) self.mock_novaclient.images = Mock(**{ 'list.return_value': self.image_list}) self.mock_novaclient.hypervisors = Mock(**{ 'list.return_value': self.hyp_list}) self.server_list = [] self.server_list.append(Server( name=randomword(), flavor_id=random.choice(self.flavor_list).id, image_id=random.choice(self.image_list).id, power_state=random.choice(power_states.keys()), metadata={'monitor': 'true'}, tenant_id=self.project_list[1].id)) # project node self.server_list.append(Server( name=randomword(), flavor_id=random.choice(self.flavor_list).id, image_id=random.choice(self.image_list).id, power_state=random.choice(power_states.keys()), metadata={}, tenant_id=self.project_list[0].id)) self.services_list = [Struct({'id': '1', 'host': 'myhost1'})] self.mock_novaclient.servers = Mock(**{ 'list.return_value': self.server_list}) self.mock_novaclient.services = Mock(**{ 'list.return_value': self.services_list, 'delete.return_value': 'Pass' }) self.mock_ksclient = MagicMock(spec=ksclient.Client) self.mock_ksclient.projects = Mock(**{ 'list.return_value': self.project_list }) self.mock_monasca_client = MagicMock(spec=msclient.Client) self.mock_monasca_client.metrics = Mock(**{ 'list_statistics.return_value': [{ "name": "some.metric", "statistics": [ ["2016-07-12T19:16:00.000Z", 31], ["2016-07-12T19:17:00.000Z", 31], ], "dimensions": {}, "columns": ["timestamp", "avg"], "id": "682bec4c92f43a52fd4f3a2855c2a026b27a063d" }], 'list_measurements.return_value': [{ "name": "some.metric", "measurements": [ ["2016-07-12T19:16:00.000Z", 10], ["2016-07-12T19:17:00.000Z", 10], ], "dimensions": {}, "columns": ["timestamp", "avg"], "id": "682bec4c92f43a52fd4f3a2855c2a026b27a063d" }] }) self.mock_get_endpoints = [{'region': None, 'url': None}] @mock.patch.object(UserGroupSvc, '_get_ks_client') @mock.patch.object(TokenHelpers, 'get_service_endpoint') @mock.patch.object(TokenHelpers, 'get_endpoints') @mock.patch.object(NovaSvc, '_get_nova_client') def test_full_instance_list(self, _mock_nova_client, _mock_endpoints, _mock_serv_end, _mock_ks_client): _mock_nova_client.return_value = self.mock_novaclient # Pretend the baremetal endpoint does not exist _mock_serv_end.return_value = None _mock_endpoints.return_value = self.mock_get_endpoints # Pretend there's one tenant called "some_tenant" _mock_ks_client.return_value = self.mock_ksclient svc = NovaSvc(BllRequest(auth_token=get_mock_token(), data={api.OPERATION: 'instance-list'})) reply = svc.handle() self.assertEqual(api.COMPLETE, reply[api.STATUS]) self.assertEqual( len(self.server_list), len(reply[api.DATA]['instances'])) @mock.patch.object(UserGroupSvc, '_get_ks_client') @mock.patch.object(TokenHelpers, 'get_service_endpoint') @mock.patch.object(TokenHelpers, 'get_endpoints') @mock.patch.object(NovaSvc, '_get_nova_client') def test_hypervisor_list(self, _mock_nova_client, _mock_endpoints, _mock_serv_end, _mock_ks_client): _mock_nova_client.return_value = self.mock_novaclient _mock_serv_end.return_value = None _mock_ks_client.return_value = self.mock_ksclient _mock_endpoints.return_value = self.mock_get_endpoints svc = NovaSvc(BllRequest(auth_token=get_mock_token(), data={api.OPERATION: 'hypervisor-list'})) # build up a list of ping statuses statuses = {} for hyp in self.hyp_list: statuses[hyp.hypervisor_hostname] = 'up' with patch('bll.plugins.service.SvcBase.call_service', return_value=statuses): reply = svc.handle() self.assertEqual(api.COMPLETE, reply[api.STATUS]) # 4 hypervisors hyp_list = reply[api.DATA] self.assertEqual(len(self.hyp_list), len(hyp_list)) known_id_list = [x.id for x in self.hyp_list] for hyp in hyp_list: self.assertTrue(hyp['hypervisor_id'] in known_id_list) self.assertEqual(hyp['ping_status'], 'up') def nova_operation(self, operation, data=None): request = { api.TARGET: 'nova', api.AUTH_TOKEN: self.token or None, api.DATA: { api.OPERATION: operation } } if data: request[api.DATA].update(data) svc = NovaSvc(bll_request=BllRequest(request)) return svc.handle() @functional('nova') def test_instance_list(self): self.token = get_token_from_env() reply = self.nova_operation('instance-list') self.assertEqual(api.COMPLETE, reply[api.STATUS]) self.assertIn('instances', reply[api.DATA]) @functional('nova') def test_service_list(self): self.token = get_token_from_env() reply = self.nova_operation('service-list') self.assertEqual(api.COMPLETE, reply[api.STATUS]) self.assertIn('total', reply[api.DATA]) self.assertIn('ok', reply[api.DATA]) self.assertIn('error', reply[api.DATA]) @functional('nova') def test_servers_list(self): self.token = get_token_from_env() end_date = date.today() start_date = end_date + timedelta(-1) reply = self.nova_operation('servers-list', { 'start_date': start_date.isoformat(), 'end_date': end_date.isoformat(), }) self.assertEqual(api.COMPLETE, reply[api.STATUS]) self.assertIn('created', reply[api.DATA]) self.assertIn('deleted', reply[api.DATA]) @functional('nova') def test_hypervisor_stats(self): self.token = get_token_from_env() reply = self.nova_operation('hypervisor-stats') self.assertEqual(api.COMPLETE, reply[api.STATUS]) self.assertIn('total', reply[api.DATA]) self.assertIn('used', reply[api.DATA]) @mock.patch.object(UserGroupSvc, '_get_ks_client') @mock.patch.object(TokenHelpers, 'get_service_endpoint') @mock.patch.object(NovaSvc, '_get_nova_client') def test_service_delete_no_input(self, _mock_nova_client, _mock_serv_end, _mock_ks_client): _mock_nova_client.return_value = self.mock_novaclient _mock_serv_end.return_value = None _mock_ks_client.return_value = self.mock_ksclient with log_level(logging.CRITICAL, 'bll'): svc = NovaSvc(BllRequest(auth_token=get_mock_token(), data={api.OPERATION: 'service-delete'})) # In production the service is called via the handle method of the # base, SvcBase, which uses pykka and catches exceptions and # returns an appropriate error response to the caller. But using # pykka in unit tests conflicts with the mock library causing any # such tests to hang as both libraries are attempting to # automagically wrap function calls in proxies. Therefore we have # to directly expect and handle the exception ourselves. with self.assertRaises(Exception): svc.handle() @mock.patch.object(UserGroupSvc, '_get_ks_client') @mock.patch.object(TokenHelpers, 'get_service_endpoint') @mock.patch.object(TokenHelpers, 'get_endpoints') @mock.patch.object(NovaSvc, '_get_nova_client') def test_service_delete_bad_input(self, _mock_nova_client, _mock_endpoints, _mock_serv_end, _mock_ks_client): _mock_nova_client.return_value = self.mock_novaclient _mock_serv_end.return_value = None _mock_ks_client.return_value = self.mock_ksclient _mock_endpoints.return_value = self.mock_get_endpoints with log_level(logging.CRITICAL, 'bll'): svc = NovaSvc(BllRequest(auth_token=get_mock_token(), operation='service-delete', data={'hostname': 'badhost'})) with self.assertRaises(Exception): svc.handle() @mock.patch.object(UserGroupSvc, '_get_ks_client') @mock.patch.object(TokenHelpers, 'get_service_endpoint') @mock.patch.object(TokenHelpers, 'get_endpoints') @mock.patch.object(NovaSvc, '_get_nova_client') def test_service_delete_good_input(self, _mock_nova_client, _mock_endpoints, _mock_serv_end, _mock_ks_client): _mock_nova_client.return_value = self.mock_novaclient _mock_serv_end.return_value = None _mock_ks_client.return_value = self.mock_ksclient _mock_endpoints.return_value = self.mock_get_endpoints svc = NovaSvc(BllRequest(auth_token=get_mock_token(), operation='service-delete', data={'hostname': 'myhost1'})) reply = svc.handle() self.assertEqual(reply[api.STATUS], api.COMPLETE) @mock.patch.object(UserGroupSvc, '_get_ks_client') @mock.patch.object(TokenHelpers, 'get_service_endpoint') @mock.patch.object(TokenHelpers, 'get_endpoints') @mock.patch.object(MonitorSvc, '_get_monasca_client') @mock.patch.object(NovaSvc, '_get_nova_client') def test_service_inc_metrics_statistics(self, _mock_nova_client, _mock_monasca_client, _mock_endpoints, _mock_serv_end, _mock_ks_client): _mock_nova_client.return_value = self.mock_novaclient _mock_serv_end.return_value = None _mock_ks_client.return_value = self.mock_ksclient _mock_monasca_client.return_value = self.mock_monasca_client _mock_endpoints.return_value = self.mock_get_endpoints req_data = { api.OPERATION: 'instance-list', 'monasca_metrics': ['some.metric'], 'monasca_data': { 'operation': 'metric_statistics' } } svc = NovaSvc(BllRequest(auth_token=get_mock_token(), data=req_data)) reply = svc.handle() self.assertEqual(reply[api.STATUS], api.COMPLETE) self.mock_monasca_client.metrics.list_statistics.assert_called_with( name='some.metric' ) self.assertIn(api.DATA, reply) data = reply[api.DATA] self.assertIn('instances', data) self.assertIsInstance(data['instances'], list) self.assertTrue(len(data['instances']) > 0) self.assertIn('metrics', data['instances'][0]) self.assertIn('some.metric', data['instances'][0]['metrics']) some_metric = data['instances'][0]['metrics']['some.metric'] self.assertEquals(some_metric[0]['statistics'][0][1], 31) @mock.patch.object(UserGroupSvc, '_get_ks_client') @mock.patch.object(TokenHelpers, 'get_service_endpoint') @mock.patch.object(TokenHelpers, 'get_endpoints') @mock.patch.object(MonitorSvc, '_get_monasca_client') @mock.patch.object(NovaSvc, '_get_nova_client') def test_service_inc_metrics_measurements(self, _mock_nova_client, _mock_monasca_client, _mock_endpoints, _mock_serv_end, _mock_ks_client): _mock_nova_client.return_value = self.mock_novaclient _mock_serv_end.return_value = None _mock_ks_client.return_value = self.mock_ksclient _mock_monasca_client.return_value = self.mock_monasca_client _mock_endpoints.return_value = self.mock_get_endpoints req_data = { api.OPERATION: 'instance-list', 'monasca_metrics': ['some.metric'], 'monasca_data': { 'operation': 'measurement_list' } } svc = NovaSvc(BllRequest(auth_token=get_mock_token(), data=req_data)) reply = svc.handle() self.assertEqual(reply[api.STATUS], api.COMPLETE) self.mock_monasca_client.metrics.list_measurements \ .assert_called_with( name='some.metric' ) self.assertIn(api.DATA, reply) data = reply[api.DATA] self.assertIn('instances', data) self.assertIsInstance(data['instances'], list) self.assertTrue(len(data['instances']) > 0) self.assertIn('metrics', data['instances'][0]) self.assertIn('some.metric', data['instances'][0]['metrics']) some_metric = data['instances'][0]['metrics']['some.metric'] self.assertEquals(some_metric[0]['measurements'][0][1], 10) @mock.patch.object(UserGroupSvc, '_get_ks_client') @mock.patch.object(TokenHelpers, 'get_service_endpoint') @mock.patch.object(TokenHelpers, 'get_endpoints') @mock.patch.object(MonitorSvc, '_get_monasca_client') @mock.patch.object(NovaSvc, '_get_nova_client') def test_service_inc_metrics_dimension_prop(self, _mock_nova_client, _mock_monasca_client, _mock_endpoints, _mock_serv_end, _mock_ks_client): _mock_nova_client.return_value = self.mock_novaclient _mock_serv_end.return_value = None _mock_ks_client.return_value = self.mock_ksclient _mock_monasca_client.return_value = self.mock_monasca_client _mock_endpoints.return_value = self.mock_get_endpoints req_data = { api.OPERATION: 'instance-list', 'project_id': self.project_list[0].id, 'monasca_metrics': ['some.metric'], 'monasca_dimensions': { 'resource_id': { 'property': 'tenant_id' } }, 'monasca_data': { 'operation': 'metric_statistics' } } svc = NovaSvc(BllRequest(auth_token=get_mock_token(), data=req_data)) reply = svc.handle() self.assertEqual(reply[api.STATUS], api.COMPLETE) self.mock_monasca_client.metrics.list_statistics \ .assert_called_with( name='some.metric', dimensions={ 'resource_id': self.server_list[4].id } ) self.assertIn(api.DATA, reply) data = reply[api.DATA] self.assertIn('instances', data) self.assertIsInstance(data['instances'], list) self.assertTrue(len(data['instances']) > 0) self.assertIn('metrics', data['instances'][0]) self.assertIn('some.metric', data['instances'][0]['metrics']) @mock.patch.object(UserGroupSvc, '_get_ks_client') @mock.patch.object(TokenHelpers, 'get_service_endpoint') @mock.patch.object(TokenHelpers, 'get_endpoints') @mock.patch.object(MonitorSvc, '_get_monasca_client') @mock.patch.object(NovaSvc, '_get_nova_client') def test_service_inc_metrics_dimension(self, _mock_nova_client, _mock_monasca_client, _mock_endpoints, _mock_serv_end, _mock_ks_client): _mock_nova_client.return_value = self.mock_novaclient _mock_serv_end.return_value = None _mock_ks_client.return_value = self.mock_ksclient _mock_monasca_client.return_value = self.mock_monasca_client _mock_endpoints.return_value = self.mock_get_endpoints req_data = { api.OPERATION: 'instance-list', 'project_id': self.project_list[0].id, 'monasca_metrics': ['some.metric'], 'monasca_dimensions': { 'cluster': 'compute' }, 'monasca_data': { 'operation': 'metric_statistics' } } svc = NovaSvc(BllRequest(auth_token=get_mock_token(), data=req_data)) reply = svc.handle() self.assertEqual(reply[api.STATUS], api.COMPLETE) self.mock_monasca_client.metrics.list_statistics \ .assert_called_with( name='some.metric', dimensions={ 'cluster': 'compute' } ) self.assertIn(api.DATA, reply) data = reply[api.DATA] self.assertIn('instances', data) self.assertIsInstance(data['instances'], list) self.assertTrue(len(data['instances']) > 0) self.assertIn('metrics', data['instances'][0]) self.assertIn('some.metric', data['instances'][0]['metrics']) 070701000A47D2000081A40000000000000000000000015B64B4DF00007B9B000000FD0000000200000000000000000000006800000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/tests/plugins/test_objectstorage_summary_service.py# (c) Copyright 2016-2017 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC from mock import patch from bll import api from bll.api.auth_token import TokenHelpers from bll.api.request import BllRequest from bll.plugins import objectstorage_summary_service from monascaclient.v2_0.alarms import AlarmsManager from monascaclient.v2_0.metrics import MetricsManager from monascaclient.v2_0.alarm_definitions import AlarmDefinitionsManager from tests.util import TestCase MEASUREMENT_OUPUT = [{'dimensions': {'service': 'ops-console', 'cluster': 'management', 'url': 'http://192.16.66.10:9095/version.json', 'hostname': 'mycloud-ccp-mgmt-m1-clm', 'component': 'ops-console-web', 'control_plane': 'ccp', 'mount': '/dev/mqueue', 'cloud_name': 'mycloud' }, 'measurements': [['2016-08-28T22:41:15.000Z', 12.0, {}], ['2016-08-28T23:41:15.000Z', 59.0, {}] ], 'id': '3a155502224c8e83b30ee13e2adfe9d89d78e602', 'columns': ['timestamp', 'value', 'value_meta'], 'name': 'test'}] STATISTICS_OUTPUT = [{'dimensions': {'service': 'object-storage', 'cluster': 'MyCluster', 'hostname': 'MyHostname', 'mount': '/dev/mqueue' }, 'name': 'Swiftlm.Test', 'statistics': [ ['2016-08-27T23:41:15.000Z', 26.9], ['2016-07-28T07:41:15.000Z', 0.0], ['2016-07-28T11:41:15.000Z', 34.5], ['2016-07-28T19:41:15.000Z', 34.5], ['2016-08-28T23:41:15.000Z', 2.0]] } ] SPECIAL_STATISTICS_OUTPUT = [{'dimensions': {'service': 'object-storage', 'cluster': 'MyCluster', 'hostname': 'MyHostname', 'mount': '/dev/mqueue' }, 'name': 'Swiftlm.Test', 'statistics': [ ['2016-08-27T23:41:15.000Z', 26.9], ['2016-07-28T04:41:15.000Z', 2.0], ['2016-07-28T07:41:15.000Z', 0.0], ['2016-07-28T11:41:15.000Z', 34.5], ['2016-07-28T15:41:15.000Z', 0.0], ['2016-07-28T19:41:15.000Z', 34.5], ['2016-08-28T23:41:15.000Z', 2.0]] } ] ALARM_DEFINITION_SHOW_OUTPUT = {'description': 'Alarms', 'id': '38b3c2b7-efe6', 'name': 'Disk Usage', 'severity': 'LOW' } ALARM_LIST_OUTPUT = [{'state': 'OK', 'alarm_definition': {'severity': 'LOW', 'id': '38b3c2b7-efe6', 'name': 'Disk Usage'}, 'id': 'ff43aacc-a5db-4f6f-a5d3-44f9cce8c713'}, {'state': 'ALARM', 'alarm_definition': {'severity': 'LOW', 'id': '38b3c2b7-efe7', 'name': 'Memory Usage'}, 'id': 'ff43aacc-a5db-4f6f-a5d3-44f9cce8c714'}, {'state': 'ALARM', 'alarm_definition': {'severity': 'HIGH', 'id': '38b3c2b7-efe8', 'name': 'Latency Usage'}, 'id': 'ff43aacc-a5db-4f6f-a5d3-44f9cce8c715'} ] ALARM_SHOW_OUTPUT = {'state': "ALARM", 'alarm_definition': {'severity': 'HIGH', 'id': '38b3c2b7-efe8', 'name': 'Latency Usage'}, 'status': 'CRITICAL', 'id': 'ff43aacc-a5db-4f6f-a5d3-44f9cce8c715' } CALL_SERVICE_OUTPUT = {'ccp:cluster1': ['standard-ccp-c1-m1-mgmt', 'standard-ccp-c1-m2-mgmt', 'standard-ccp-c1-m3-mgmt'], 'ccp:cluster2': ['standard-ccp-c1-m1-mgmt', 'standard-ccp-c1-m2-mgmt', 'standard-ccp-c1-m3-mgmt']} ALARM_COUNT_OUTPUT = {"counts": [[5, "OK", "LOW"], [6, "ALARM", "CRITICAL"], [2, "ALARM", "HIGH"], [3, "ALARM", "LOW"], [7, "UNDETERMINED", "MED"]]} METRIC_LIST_OUTPUT = [{"id": "0000f6b224259cf215505608edf6f10d7a3a273d", "name": "swiftlm.systems.check_mounts", "dimensions": {"cluster": "MyCluster", "hostname": "Myhostname", "service": "object-storage", "mount": "/dev/mqueue"} }] class Object_Storage_Data(): def data_without_cluster_card(self): return {'end_time': '2016-08-28T23:41:15Z', 'interval': '1', 'period': '3600'} def data_without_cluster_graph(self): return {'end_time': '2016-08-28T23:41:15Z', 'interval': '5', 'period': '3600'} def data_with_cluster_card(self): return {'end_time': '2016-08-28T23:41:15Z', 'interval': '1', 'period': '3600', 'cluster': 'MyCluster', 'hostname': 'MyHostname'} def data_with_cluster_graph(self): return {'end_time': '2016-08-28T23:41:15Z', 'interval': '5', 'period': '3600', 'cluster': 'MyCluster', 'hostname': 'Myhostname'} def data_only_node(self): return {'cluster': 'MyCluster', 'hostname': 'Myhostname'} class TestObjectStorageSummarySvc(TestCase): def setUp(self): self.inst = Object_Storage_Data() def test_memory_card(self): data = self.inst.data_with_cluster_card() expected_output = {'Swiftlm.Test': 2.0} self.common_handler(operation='memory', data=data, expected_output=expected_output) def test_storage_card(self): data = self.inst.data_with_cluster_card() expected_output = {'Swiftlm.Test': 2.0} self.common_handler(operation='storage', data=data, expected_output=expected_output) def test_load_average_donut(self): data = self.inst.data_with_cluster_card() expected_output = {'Swiftlm.Test': 2.0} self.common_handler(operation='load_average_donut', data=data, expected_output=expected_output) def test_time_to_replicate_card(self): data = self.inst.data_without_cluster_card() expected_output = {'Swiftlm.Test': 2.0} self.common_handler(operation='time_to_replicate', data=data, expected_output=expected_output) def test_time_to_replicate_graph(self): data = self.inst.data_without_cluster_graph() expected_output = {'Swiftlm.Test': [['2016-08-27T23:41:15.000Z', 26.9], ['2016-07-28T07:41:15.000Z', 0.0], ['2016-07-28T11:41:15.000Z', 34.5], ['2016-07-28T19:41:15.000Z', 34.5], ['2016-08-28T22:41:15Z', -1], ['2016-08-28T23:41:15.000Z', 2.0] ]} self.common_handler(operation='time_to_replicate', data=data, expected_output=expected_output) def test_oldest_replication_completion_card(self): data = self.inst.data_without_cluster_card() expected_output = {'Swiftlm.Test': 2.0} self.common_handler( operation='oldest_replication_completion', data=data, expected_output=expected_output) def test_oldest_replication_completion_graph(self): data = self.inst.data_without_cluster_graph() expected_output = {'Swiftlm.Test': [['2016-08-27T23:41:15.000Z', 26.9], ['2016-07-28T07:41:15.000Z', 0.0], ['2016-07-28T11:41:15.000Z', 34.5], ['2016-07-28T19:41:15.000Z', 34.5], ['2016-08-28T22:41:15Z', -1], ['2016-08-28T23:41:15.000Z', 2.0] ]} self.common_handler( operation='oldest_replication_completion', data=data, expected_output=expected_output) def test_current_capacity_card(self): data = self.inst.data_without_cluster_card() expected_output = {'Swiftlm.Test': 2.0} self.common_handler(operation='current_capacity', data=data, expected_output=expected_output) def test_current_capacity_graph(self): data = self.inst.data_without_cluster_graph() expected_output = {'Swiftlm.Test': [['2016-08-27T23:41:15.000Z', 26.9], ['2016-07-28T07:41:15.000Z', 0.0], ['2016-07-28T11:41:15.000Z', 34.5], ['2016-07-28T19:41:15.000Z', 34.5], ['2016-08-28T22:41:15Z', -1], ['2016-08-28T23:41:15.000Z', 2.0]]} self.common_handler(operation='current_capacity', data=data, expected_output=expected_output) def test_filesystem_utilization_card(self): data = self.inst.data_with_cluster_card() expected_output = {'Swiftlm.Test': 2.0} self.common_handler(operation='filesystem_utilization', data=data, expected_output=expected_output) def test_latency_healthcheck_card(self): data = self.inst.data_without_cluster_card() expected_output = {'Swiftlm.Test': 2.0} self.common_handler(operation='latency_healthcheck', data=data, expected_output=expected_output) def test_latency_healthcheck_graph(self): data = self.inst.data_without_cluster_graph() expected_output = {'Swiftlm.Test': [['2016-08-27T23:41:15.000Z', 26.9], ['2016-07-28T07:41:15.000Z', 0.0], ['2016-07-28T11:41:15.000Z', 34.5], ['2016-07-28T19:41:15.000Z', 34.5], ['2016-08-28T22:41:15Z', -1], ['2016-08-28T23:41:15.000Z', 2.0] ]} self.common_handler(operation='latency_healthcheck', data=data, expected_output=expected_output) def test_latency_operational_card(self): data = self.inst.data_without_cluster_card() expected_output = {'Swiftlm.Test': 2.0} self.common_handler(operation='latency_operational', data=data, expected_output=expected_output) def test_latency_operational_graph(self): data = self.inst.data_without_cluster_graph() expected_output = {'Swiftlm.Test': [['2016-08-27T23:41:15.000Z', 26.9], ['2016-07-28T07:41:15.000Z', 0.0], ['2016-07-28T11:41:15.000Z', 34.5], ['2016-07-28T19:41:15.000Z', 34.5], ['2016-08-28T22:41:15Z', -1], ['2016-08-28T23:41:15.000Z', 2.0] ]} self.common_handler(operation='latency_operational', data=data, expected_output=expected_output) def test_async_pending_card(self): data = self.inst.data_without_cluster_card() expected_output = {'Swiftlm.Test': 2.0} self.common_handler(operation='async_pending', data=data, expected_output=expected_output) def test_async_pending_graph(self): data = self.inst.data_without_cluster_graph() expected_output = {'Swiftlm.Test': [['2016-08-27T23:41:15.000Z', 26.9], ['2016-07-28T07:41:15.000Z', 0.0], ['2016-07-28T11:41:15.000Z', 34.5], ['2016-07-28T19:41:15.000Z', 34.5], ['2016-08-28T22:41:15Z', -1], ['2016-08-28T23:41:15.000Z', 2.0] ]} self.common_handler(operation='async_pending', data=data, expected_output=expected_output) def test_alarms(self): data = self.inst.data_without_cluster_card() expected_output = {'counts': [[5, 'OK', 'LOW'], [6, 'ALARM', 'CRITICAL'], [2, 'ALARM', 'HIGH'], [3, 'ALARM', 'LOW'], [7, 'UNDETERMINED', 'MED'] ]} self.common_handler(operation='alarms', data=data, expected_output=expected_output) def test_mount_status(self): data = self.inst.data_with_cluster_card() expected_output = {'total_mount_point': 1, 'mount_status': {'mounted': 0, 'unmounted': 1}} self.common_handler(operation='mount_status', data=data, expected_output=expected_output) def test_service_availability_card(self): data = self.inst.data_without_cluster_card() expected_output = {'Swiftlm.Test': 2.0} self.common_handler(operation='service_availability', data=data, expected_output=expected_output) def test_service_availability_graph(self): data = self.inst.data_without_cluster_graph() expected_output = {'Swiftlm.Test': [['2016-08-27T23:41:15.000Z', 26.9], ['2016-07-28T07:41:15.000Z', 0.0], ['2016-07-28T11:41:15.000Z', 34.5], ['2016-07-28T19:41:15.000Z', 34.5], ['2016-08-28T22:41:15Z', -1], ['2016-08-28T23:41:15.000Z', 2.0] ]} self.common_handler(operation='service_availability', data=data, expected_output=expected_output) def test_load_average_card(self): data = self.inst.data_without_cluster_card() expected_output = {'Swiftlm.Test': 2.0} self.common_handler(operation='load_average', data=data, expected_output=expected_output) def test_load_avaergae_graph(self): data = self.inst.data_without_cluster_graph() expected_output = {'Swiftlm.Test': [['2016-08-27T23:41:15.000Z', 26.9], ['2016-07-28T07:41:15.000Z', 0.0], ['2016-07-28T11:41:15.000Z', 34.5], ['2016-07-28T19:41:15.000Z', 34.5], ['2016-08-28T22:41:15Z', -1], ['2016-08-28T23:41:15.000Z', 2.0] ]} self.common_handler(operation='load_average', data=data, expected_output=expected_output) def test_file_systems(self): data = self.inst.data_with_cluster_card() expected_output = {'/dev/mqueue': {'Swiftlm.Test': 2.0}} self.common_handler(operation='file_systems', data=data, expected_output=expected_output) def test_rate_of_change_card(self): data = self.inst.data_with_cluster_card() expected_output = [-32.5] self.common_handler(operation='rate_of_change', data=data, expected_output=expected_output) def test_rate_of_change_graph(self): data = self.inst.data_with_cluster_graph() expected_output = [['2016-08-27T23:41:15.000Z', -26], ['2016-07-28T07:41:15.000Z', 34], ['2016-07-28T11:41:15.000Z', 0], ['2016-07-28T19:41:15.000Z', -32], ['2016-08-28T22:41:15Z', -1], ['2016-08-28T23:41:15Z', -1]] self.common_handler(operation='rate_of_change', data=data, expected_output=expected_output) def test_heat_map_cpu_load_average(self): data = self.inst.data_without_cluster_card() self.common_handler(operation='heat_map_cpu_load_average', data=data, expected_output=[]) def test_alarm_description(self): data = self.inst.data_with_cluster_card() expected_output = {'ff43aacc-a5db-4f6f-a5d3-44f9cce8c713': {'status': 'CRITICAL', 'state': 'ALARM', 'description': 'Alarms', 'alarm_definition_id': '38b3c2b7-efe8', 'name': 'Latency Usage', 'severity': 'HIGH'}, 'ff43aacc-a5db-4f6f-a5d3-44f9cce8c715': {'status': 'CRITICAL', 'state': 'ALARM', 'description': 'Alarms', 'alarm_definition_id': '38b3c2b7-efe8', 'name': 'Latency Usage', 'severity': 'HIGH'}, 'ff43aacc-a5db-4f6f-a5d3-44f9cce8c714': {'status': 'CRITICAL', 'state': 'ALARM', 'description': 'Alarms', 'alarm_definition_id': '38b3c2b7-efe8', 'name': 'Latency Usage', 'severity': 'HIGH'}} self.common_handler(operation='alarm_description', data=data, expected_output=expected_output) def test_heat_map_utilization_focused_inventory(self): data = self.inst.data_without_cluster_card() self.common_handler( operation='heat_map_utilization_focused_inventory', data=data, expected_output=[]) @patch.object(AlarmsManager, 'count') @patch.object(objectstorage_summary_service.ObjectStorageSummarySvc, 'call_service') @patch.object(TokenHelpers, 'get_service_endpoint') @patch.object(TokenHelpers, 'get_token_for_project') def test_node_state(self, mock_get_token_for_project, mock_get_service_endpoint, mock_call_service, mock_alarm_count): mock_get_token_for_project.return_value = "admin" mock_get_service_endpoint.return_value = "http://localhost:8070/v2.0" mock_call_service.return_value = {'ccp:cluster1': ['standard-ccp-c1-m1-mgmt', 'standard-ccp-c1-m2-mgmt', 'standard-ccp-c1-m3-mgmt'], 'ccp:cluster2': ['standard-ccp-c1-m1-mgmt', 'standard-ccp-c1-m2-mgmt', 'standard-ccp-c1-m3-mgmt']} mock_alarm_count.return_value = {"counts": [[5, "OK", "LOW"], [6, "ALARM", "CRITICAL"], [2, "ALARM", "HIGH"], [3, "ALARM", "LOW"], [7, "UNDETERMINED", "MED"]]} request = { api.TARGET: 'objectstorage_summary_service', api.ACTION: 'GET', api.AUTH_TOKEN: 'unused', api.DATA: { api.OPERATION: "node_state", api.DATA: None } } svc = objectstorage_summary_service.ObjectStorageSummarySvc( bll_request=BllRequest(request)) reply = svc.handle() self.assertEqual(reply['status'], api.STATUS_INPROGRESS) @patch.object(AlarmsManager, 'count') @patch.object(objectstorage_summary_service.ObjectStorageSummarySvc, 'call_service') @patch.object(TokenHelpers, 'get_service_endpoint') @patch.object(TokenHelpers, 'get_token_for_project') def test_health_focused(self, mock_get_token_for_project, mock_get_service_endpoint, mock_call_service, mock_alarm_count): mock_get_token_for_project.return_value = "admin" mock_get_service_endpoint.return_value = "http://localhost:8070/v2.0" mock_call_service.return_value = {'ccp:cluster1': ['standard-ccp-c1-m1-mgmt', 'standard-ccp-c1-m2-mgmt', 'standard-ccp-c1-m3-mgmt'], 'ccp:cluster2': ['standard-ccp-c1-m1-mgmt', 'standard-ccp-c1-m2-mgmt', 'standard-ccp-c1-m3-mgmt']} mock_alarm_count.return_value = {"counts": [[5, "OK", "LOW"], [6, "OK", "HIGH"]]} request = { api.TARGET: 'objectstorage_summary_service', api.ACTION: 'GET', api.AUTH_TOKEN: 'unused', api.DATA: { api.OPERATION: "health_focused", api.DATA: None } } svc = objectstorage_summary_service.ObjectStorageSummarySvc( bll_request=BllRequest(request)) reply = svc.handle() self.assertEqual(reply['status'], api.STATUS_INPROGRESS) @patch.object(objectstorage_summary_service.ObjectStorageSummarySvc, 'call_service') @patch.object(AlarmsManager, 'list') @patch.object(AlarmsManager, 'get') @patch.object(AlarmDefinitionsManager, 'get') @patch.object(AlarmsManager, 'count') @patch.object(MetricsManager, 'list') @patch.object(MetricsManager, 'list_statistics') @patch.object(MetricsManager, 'list_measurements') @patch.object(TokenHelpers, 'get_service_endpoint') @patch.object(TokenHelpers, 'get_token_for_project') def common_handler(self, mock_get_token_for_project, mock_get_service_endpoint, mock_get_measurement_list, mock_get_statistics_list, mock_get_list, mock_get_alarm_count, mock_get_alarm_definition_get, mock_get_alarm_get, mock_get_alarm_list, mock_call_service, operation, data, expected_output): mock_get_token_for_project.return_value = "admin" mock_get_service_endpoint.return_value = "http://localhost:8070/v2.0" mock_get_measurement_list.return_value = MEASUREMENT_OUPUT mock_get_statistics_list.return_value = STATISTICS_OUTPUT mock_get_list.return_value = METRIC_LIST_OUTPUT mock_get_alarm_count.return_value = ALARM_COUNT_OUTPUT mock_get_alarm_definition_get.return_value = \ ALARM_DEFINITION_SHOW_OUTPUT mock_get_alarm_get.return_value = ALARM_SHOW_OUTPUT mock_get_alarm_list.return_value = ALARM_LIST_OUTPUT mock_call_service.return_value = CALL_SERVICE_OUTPUT request = { api.TARGET: 'objectstorage_summary_service', api.ACTION: 'GET', api.AUTH_TOKEN: 'unused', api.DATA: { api.OPERATION: operation, api.DATA: data } } svc = objectstorage_summary_service.ObjectStorageSummarySvc( bll_request=BllRequest(request)) reply = svc.handle() expected = expected_output self.assertEqual(reply[api.DATA], expected) def test_project_capacity_metric_card_selected_project(self): data = {"end_time": "2016-07-15T23:00:00Z", "interval": 1, "period": 3600, "id": "123"} operation = "project_capacity" self.project_common_handler(operation=operation, data=data) def test_project_capacity_metric_card_all_project(self): data = {"end_time": "2016-07-15T23:00:00Z", "interval": 1, "period": 3600, "id": "all"} operation = "project_capacity" self.project_common_handler(operation=operation, data=data) def test_project_capacity_time_series_all_project(self): data = {"end_time": "2016-07-15T23:00:00Z", "interval": 24, "period": 3600, "id": "all"} operation = "project_capacity" self.project_common_handler(operation=operation, data=data) def test_project_capacity_time_series_selected_project(self): data = {"end_time": "2016-07-15T23:00:00Z", "interval": 24, "period": 3600, "id": "123"} operation = "project_capacity" self.project_common_handler(operation=operation, data=data) def test_project_capacity_roc_metric_card_selected_project(self): data = {"end_time": "2016-07-15T23:00:00Z", "interval": 2, "period": 3600, "id": "123"} operation = "project_capacity_roc" self.project_common_handler(operation=operation, data=data) def test_project_capacity_roc_metric_card_all_project(self): data = {"end_time": "2016-07-15T23:00:00Z", "interval": 2, "period": 3600, "id": "all"} operation = "project_capacity_roc" self.project_common_handler(operation=operation, data=data) def test_project_capacity_roc_time_series_all_project(self): data = {"end_time": "2016-07-15T23:00:00Z", "interval": 24, "period": 3600, "id": "all"} operation = "project_capacity_roc" self.project_common_handler(operation=operation, data=data) def test_project_capacity_roc_time_series_selected_project(self): data = {"end_time": "2016-07-15T23:00:00Z", "interval": 24, "period": 3600, "id": "123"} operation = "project_capacity_roc" self.project_common_handler(operation=operation, data=data) def test_topten_project_capacity(self): data = {"end_time": "2016-07-15T23:00:00Z", "interval": 6, "period": 3600} operation = "topten_project_capacity" self.project_common_handler(operation=operation, data=data) @patch.object(MetricsManager, 'list_statistics') @patch.object(objectstorage_summary_service.ObjectStorageSummarySvc, 'call_service') @patch.object(TokenHelpers, 'get_service_endpoint') @patch.object(TokenHelpers, 'get_token_for_project') def project_common_handler(self, mock_get_token_for_project, mock_get_service_endpoint, mock_call_service, mock_list_statistics, operation, data): mock_get_token_for_project.return_value = "admin" mock_get_service_endpoint.return_value = "http://localhost:8070/v2.0" mock_call_service.return_value = [ {"id": "34c037934d852ea7", "name": "backup"}, {"id": "2e3a733b4559c20f", "name": "demo"} ] mock_list_statistics.return_value = [ {"dimensions": {"user_id": "None", "cloud_name": "standard", "region": "None", "resource_id": "62d256ae5f1748dab8fedf8ebdf4b802", "control_plane": "ccp", "cluster": "cluster1", "datasource": "ceilometer", "project_id": "62d256ae5f1748dab8fedf8ebdf4b802", "type": "gauge", "unit": "B", "source": "openstack"}, "statistics": [["2016-07-15T23:00:00.000Z", 300], ["2016-07-16T00:00:00.000Z", 450], ["2016-07-16T01:00:00.000Z", 250], ["2016-07-16T02:00:00.000Z", 500], ["2016-07-16T03:00:00.000Z", 600] ] }] request = { api.TARGET: 'objectstorage_summary_service', api.ACTION: 'GET', api.AUTH_TOKEN: 'unused', api.DATA: { api.OPERATION: operation, api.DATA: data, } } svc = objectstorage_summary_service.ObjectStorageSummarySvc( bll_request=BllRequest(request)) reply = svc.handle() self.assertEqual(reply['status'], api.STATUS_INPROGRESS) 070701000A47D1000081A40000000000000000000000015B64B4DF00000CBC000000FD0000000200000000000000000000005600000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/tests/plugins/test_preferences.py# (c) Copyright 2015-2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC from bll import api from bll.api.request import BllRequest from bll.plugins.service import SvcBase import logging import random from tests.util import functional, TestCase, randomword, randomidentifier,\ get_mock_token, log_level @functional('mysql') class TestPreferencesSvc(TestCase): def handle(self, action, user='test', prefs=None): req_dict = { api.TARGET: 'preferences', api.AUTH_TOKEN: get_mock_token(), api.ACTION: action } req_dict[api.DATA] = {"user": user} if prefs: req_dict[api.DATA]["prefs"] = prefs return SvcBase.spawn_service(BllRequest(req_dict)) def test_crud_operations(self): user = randomword() prefs = self.create_random_dict() # GET preferences for a non-existent user with log_level(logging.CRITICAL, 'bll.plugins.service'): reply = self.handle('GET', user) self.assertEqual('error', reply[api.STATUS]) try: reply = self.handle('POST', user, prefs) self.assertEqual('complete', reply[api.STATUS]) reply = self.handle('GET', user) self.assertEqual('complete', reply[api.STATUS]) self.assertEqual(prefs, reply[api.DATA]) prefs = self.create_random_dict() reply = self.handle('PUT', user, prefs) self.assertEqual('complete', reply[api.STATUS]) reply = self.handle('GET', user) self.assertEqual('complete', reply[api.STATUS]) self.assertEqual(prefs, reply[api.DATA]) finally: reply = self.handle('DELETE', user) self.assertEqual('complete', reply[api.STATUS]) with log_level(logging.CRITICAL, 'bll.plugins.service'): reply = self.handle('GET', user) self.assertEqual('error', reply[api.STATUS]) def test_get_from_missing_user(self): # GET preferences for a non-existent user with log_level(logging.CRITICAL, 'bll.plugins.service'): reply = self.handle('GET', randomword()) self.assertEqual('error', reply[api.STATUS]) def test_put_to_missing_user(self): # PUT preferences for a non-existent user with log_level(logging.CRITICAL, 'bll.plugins.service'): reply = self.handle('PUT', randomword(), randomword()) self.assertEqual('error', reply[api.STATUS]) def test_delete_of_missing_user(self): # DELETE preferences for a non-existent user should be ok reply = self.handle('DELETE', randomword()) self.assertEqual('complete', reply[api.STATUS]) def create_random_dict(self): # build a complicated, nested dictionary my_dict = {} my_dict[randomidentifier()] = randomword() nested_dict = {} nested_dict[randomidentifier()] = randomword() my_dict['dict'] = nested_dict nested_array = [] nested_array.append(randomword()) nested_array.append(random.random()) nested_array.append(random.randint(0, 1000)) my_dict['array'] = nested_array return my_dict 070701000A47D8000081A40000000000000000000000015B64B4DF00000E47000000FD0000000200000000000000000000005800000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/tests/plugins/test_region_client.py# (c) Copyright 2016-2017 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC from bll.api.auth_token import TokenHelpers from bll.plugins.region_client import RegionClient import mock from tests.util import TestCase, randomidentifier, get_mock_token, randomurl class TestRegionClient(TestCase): @mock.patch.object(TokenHelpers, 'get_endpoints') def test_single_region(self, _mock_endpoints): """ In a single-region setup, a call to get one client must return a list containing just the single client, and calling them in either order (the list first then the single, or vise-versa) must result in just a single call to the create function. """ _mock_endpoints.return_value = [{'region': randomidentifier(), 'url': randomurl()}] # # Get all clients first, then get just a single client # create_func = mock.Mock() client = RegionClient(randomidentifier(), create_func, get_mock_token(), randomidentifier()) client_list = list(client.get_clients()) self.assertEqual(1, create_func.call_count) single_client = client.get_client() self.assertEqual(1, create_func.call_count) self.assertEqual(1, len(client_list)) self.assertEqual(single_client, client_list[0]) # # Get one client first, then get all clients # create_func = mock.Mock() client = RegionClient(randomidentifier(), create_func, get_mock_token(), randomidentifier()) single_client = client.get_client() self.assertEqual(1, create_func.call_count) client_list = list(client.get_clients()) self.assertEqual(1, create_func.call_count) self.assertEqual(1, len(client_list)) self.assertEqual(single_client, client_list[0]) @mock.patch.object(TokenHelpers, 'get_endpoints') def test_two_regions(self, _mock_endpoints): """ In a two-region setup, a call to get one client must return just one client, while a call to get all clients should return a list with that client and one other. Also, calling them in either order (list first or single client first) must generate no unnecessary calls to the create function. """ _mock_endpoints.return_value = [ {'region': randomidentifier(), 'url': randomurl()}, {'region': randomidentifier(), 'url': randomurl()}] # # Get all clients first, then get just a single client # create_func = mock.Mock() client = RegionClient(randomidentifier(), create_func, get_mock_token(), randomidentifier()) client_list = list(client.get_clients()) self.assertEqual(2, create_func.call_count) single_client = client.get_client() self.assertEqual(2, create_func.call_count) self.assertEqual(2, len(client_list)) self.assertIn(single_client, client_list) # # Get single client first, then get all clients # create_func = mock.Mock() client = RegionClient(randomidentifier(), create_func, get_mock_token(), randomidentifier()) single_client = client.get_client() self.assertEqual(1, create_func.call_count) client_list = list(client.get_clients()) self.assertEqual(2, create_func.call_count) self.assertEqual(2, len(client_list)) self.assertIn(single_client, client_list) 070701000A47D9000081A40000000000000000000000015B64B4DF000028FE000000FD0000000200000000000000000000005D00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/tests/plugins/test_service_collection.py# (c) Copyright 2015-2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC import mock import time from bll.plugins.service import SvcBase, expose from tests.util import TestCase, randomword from bll.api.request import BllRequest from bll.common.job_status import get_job_status from bll import api class TestSvcCollection(TestCase): def test_spawn_short(self): bll_request = { api.TARGET: 'general', api.DATA: { api.OPERATION: 'null' } } reply = SvcBase.spawn_service(BllRequest(bll_request)) self.assertEqual(reply[api.STATUS], api.COMPLETE) self.assertIsNotNone(reply.get(api.TXN_ID)) self.assertIn(api.DURATION, reply) self.assertIn(api.ENDTIME, reply) self.assertIn(api.STARTTIME, reply) def test_spawn_long(self): pauses = 5 pause_sec = 0.1 bll_request = { api.TARGET: 'general', api.DATA: { api.OPERATION: 'progress', 'pause_sec': pause_sec, 'num_pauses': pauses, } } reply = SvcBase.spawn_service(BllRequest(bll_request)) txn_id = reply.get(api.TXN_ID) # test the incomplete reply self.assertIn(api.TXN_ID, reply) self.assertIn(api.PROGRESS, reply) self.assertIn(api.POLLING_INTERVAL, reply) self.assertIn(api.STARTTIME, reply) self.assertIn(api.STATUS, reply) self.assertEqual(api.STATUS_INPROGRESS, reply[api.STATUS]) # wait a little extra before the first poll time.sleep(0.1 + pause_sec) # read the job status update reply = get_job_status(txn_id) self.assertEqual(reply[api.TXN_ID], txn_id) self.assertIn(api.STARTTIME, reply) self.assertIn(api.STATUS, reply) self.assertEqual(api.STATUS_INPROGRESS, reply[api.STATUS]) percent = reply[api.PROGRESS][api.PERCENT_COMPLETE] self.assertGreater(percent, 0) self.assertLess(percent, 100) while reply.get(api.STATUS) != api.COMPLETE: time.sleep(pause_sec) reply = get_job_status(txn_id) self.assertEqual(reply[api.TXN_ID], txn_id) self.assertIn(api.DURATION, reply) self.assertIn(api.ENDTIME, reply) self.assertIn(api.STARTTIME, reply) self.assertIn(api.STATUS, reply) self.assertEqual(reply[api.STATUS], api.COMPLETE) self.assertEqual(reply[api.PROGRESS][api.PERCENT_COMPLETE], 100) def test_call_service(self): # Test an sync service that calls another sync service via # SvcBase.call_service, and fails bll_request = { api.TARGET: 'composite', api.DATA: {api.OPERATION: 'composite'} } # load the composite plugin reply = SvcBase.spawn_service(BllRequest(bll_request)) self.assertIn(api.TXN_ID, reply) self.assertIsNotNone(reply.get(api.TXN_ID)) self.assertIn(api.DURATION, reply) self.assertIn(api.ENDTIME, reply) self.assertIn(api.STARTTIME, reply) self.assertIn(api.STATUS, reply) self.assertEqual(api.COMPLETE, reply[api.STATUS]) self.assertEqual(len(reply[api.DATA]), 2) def test_call_service_fail(self): # Test an sync service that calls another sync service via # SvcBase.call_service, and fails bll_request = { api.TARGET: 'composite', api.DATA: {api.OPERATION: 'fail'} } # load the composite plugin reply = SvcBase.spawn_service(BllRequest(bll_request)) self.assertEqual(api.STATUS_ERROR, reply[api.STATUS]) def test_call_service_async_indirect(self): # Test an async service that calls another async service via # SvcBase.call_service_async pauses = 2 pause_sec = 0.1 request = BllRequest(target='composite-async', operation='progress', data={ 'pause_sec': pause_sec, 'num_pauses': pauses, }) reply = SvcBase.spawn_service(request) txn_id = reply.get(api.TXN_ID) while reply.get(api.STATUS) != api.COMPLETE: time.sleep(pause_sec) reply = get_job_status(txn_id) self.assertIn(api.TXN_ID, reply) self.assertIsNotNone(reply.get(api.TXN_ID)) self.assertIn(api.DURATION, reply) self.assertIn(api.ENDTIME, reply) self.assertIn(api.STARTTIME, reply) self.assertIn(api.STATUS, reply) self.assertEqual(api.COMPLETE, reply[api.STATUS]) def test_call_service_async_indirect_fail(self): # Test an async service that calls another async service via # SvcBase.call_async, and which fails pause_sec = 0.1 request = BllRequest(target='composite-async', operation='fail') reply = SvcBase.spawn_service(request) txn_id = reply.get(api.TXN_ID) while reply.get(api.STATUS) in (api.STATUS_INPROGRESS, api.STATUS_NOT_FOUND): time.sleep(pause_sec) reply = get_job_status(txn_id) self.assertIn(api.TXN_ID, reply) self.assertIsNotNone(reply.get(api.TXN_ID)) self.assertEqual(api.STATUS_ERROR, reply[api.STATUS]) def test_call_service_async_error_propagated(self): # Verify that an error returned by the called function is propagated # as an exception to the caller of call_service_async class Foo(SvcBase): @expose(is_long=True) def bar(self): try: self.call_service_async(target="general", operation="errorcomplete", polling_interval=0.1) self.response.error("Should have thrown an exception") except Exception as e: if str(e).startswith("some error happened"): self.response.complete() else: self.response.error("Wrong type of exception") svc = Foo(BllRequest(operation="bar")) reply = svc.complete() self.assertEquals(api.COMPLETE, reply[api.STATUS]) def test_call_service_async_failure_propagated(self): # Verify that an exception returned by the called function is # propagated as an exception to the caller of call_service_async class Foo(SvcBase): @expose(is_long=True) def bar(self): try: self.call_service_async(target="general", operation="failcomplete", polling_interval=0.1) self.response.error("Should have thrown an exception") except Exception as e: if str(e).startswith("Intentional"): self.response.complete() else: self.response.error("Wrong type of exception") svc = Foo(BllRequest(operation="bar")) reply = svc.complete() self.assertEquals(api.COMPLETE, reply[api.STATUS]) def test_call_service_async_return_data(self): # Verify that the data returned from the called function is the # return value of call_service_async class Foo(SvcBase): @expose(is_long=True) def bar(self): msg = randomword() reply = self.call_service_async(target="general", operation="echo_slow", message=msg, polling_interval=0.1) if reply != msg: self.response.error("Did not receive data") svc = Foo(BllRequest(operation="bar")) reply = svc.complete() self.assertEquals(api.COMPLETE, reply[api.STATUS]) def test_call_service_async_timeout(self): class Foo(SvcBase): @expose(is_long=True) def bar(self): req = BllRequest(target="general", operation="progress", pause_sec=0.1, num_pauses=5) try: self.call_service_async(req, polling_interval=0.1, max_polls=2) self.response.error("Should have timed out") except Exception as e: if str(e).startswith("Timed out"): self.response.complete() else: self.response.error("Wrong type of exception") svc = Foo(BllRequest(operation="bar")) reply = svc.complete() self.assertEquals(api.COMPLETE, reply[api.STATUS]) def test_call_service_async_with_progress(self): class Foo(SvcBase): @expose(is_long=True) def bar(self): req = BllRequest(target="general", operation="progress", pause_sec=.1, num_pauses=5) self.call_service_async(req, polling_interval=0.05, offset=50, scale=0.5) self.response.complete() return self.response svc = Foo(BllRequest(operation="bar")) svc.update_job_status = mock.Mock() reply = svc.complete() self.assertEquals(api.COMPLETE, reply[api.STATUS]) # Note that any_order permits extra calls to have been made, which # will happen since there may be multiple calls in a row with the # same percentage_complete svc.update_job_status.assert_has_calls([ mock.call(percentage_complete=60.0), mock.call(percentage_complete=70.0), mock.call(percentage_complete=80.0), mock.call(percentage_complete=90.0), mock.call(percentage_complete=100.0)], any_order=True) 070701000A47D0000081A40000000000000000000000015B64B4DF000020CA000000FD0000000200000000000000000000005800000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/tests/plugins/test_users_service.py# (c) Copyright 2015-2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC import os from bll.common.util import get_service_tenant_name from tests.util import functional, get_token_from_env, TestCase, \ randomidentifier from bll import api from bll.plugins.user_group_service import UserGroupSvc from bll.api.request import BllRequest @functional('keystone') class TestUserGroupSvc(TestCase): def setUp(self): self.token = get_token_from_env() def tearDown(self): self._user_cleanup() def test_get_user_list(self): users = self._get_user_list() me = os.getenv("OS_USERNAME").upper() found = False for user in users: if user['username'].upper() == me: found = True break self.assertTrue(found) def test_add_user_list(self): username = self.newuser() bll_request = { api.AUTH_TOKEN: self.token, 'target': 'user_group', 'data': { 'operation': 'user_add', 'username': username, 'password': 'password', 'email': 'test@email.com', 'api_version': 'v1', 'project_name': get_service_tenant_name(), } } app_svc = UserGroupSvc(bll_request=BllRequest(bll_request)) reply = app_svc.handle() self.assertIn('status', reply) self.assertEqual('complete', reply['status']) users = self._get_user_list() user_id = None for user in users: if user['username'].upper() == username.upper(): user_id = user['user_id'] project_id = user['project_id'] break else: self.fail() user_id_list = [user_id] bll_request = { api.AUTH_TOKEN: self.token, 'target': 'user_group', 'data': { 'operation': 'users_remove', 'user_ids': user_id_list, 'password': 'password', 'email': 'test@email.com', 'project_id': project_id, 'api_version': 'v1', } } app_svc = UserGroupSvc(bll_request=BllRequest(bll_request)) reply = app_svc.handle() self.assertIn('status', reply) self.assertEqual('complete', reply['status']) # test the user is gone user_id = None for user in self._get_user_list(): if username == user['username']: user_id = user['user_id'] break if user_id is not None: self.assertTrue(1) def test_del_user_list(self): users_before = self._get_user_list() user1 = self._add_user(self.newuser(), 'tester1@test.com') user2 = self._add_user(self.newuser(), 'tester2@test.com') user_id_list = [user1, user2] project_id = None for user in self._get_user_list(): if user1 == user['user_id']: project_id = user['project_id'] break bll_request = { api.AUTH_TOKEN: self.token, 'target': 'user_group', 'data': { 'operation': 'users_remove', 'user_ids': user_id_list, 'project_id': project_id, 'api_version': 'v1', } } app_svc = UserGroupSvc(bll_request=BllRequest(bll_request)) app_svc.handle() users_after = self._get_user_list() self.assertEqual(len(users_before), len(users_after)) def test_update_user(self): user_name = self.newuser() email = 'UpdateMeEmail@email.com' update_user = self._add_user(user_name, email, project=get_service_tenant_name()) updated_user = self.newuser() updated_pass = 'NewPassword' bll_request = { api.AUTH_TOKEN: self.token, 'target': 'user_group', 'data': { 'operation': 'user_update', 'user_id': update_user, 'username': updated_user, 'email': email, 'password': updated_pass, 'api_version': 'v1', } } app_svc = UserGroupSvc(bll_request=BllRequest(bll_request)) reply = app_svc.handle() self.assertIn('status', reply) self.assertEqual('complete', reply['status']) users = self._get_user_list() found = False project_id = None for user in users: if user['username'] == updated_user: project_id = user['project_id'] found = True break if found is False: self.assertTrue(False) self._del_user(update_user, project_id) def test_backend(self): bll_request = { api.AUTH_TOKEN: self.token, 'target': 'user_group', 'data': { 'operation': 'identity_backend', } } app_svc = UserGroupSvc(bll_request=BllRequest(bll_request)) reply = app_svc.handle() self.assertIn('status', reply) self.assertEqual('complete', reply['status']) def _add_user(self, user_name, email, password='password', project=None): if project is None: project = get_service_tenant_name() bll_request = { api.AUTH_TOKEN: self.token, 'target': 'user_group', 'data': { 'operation': 'user_add', 'username': user_name, 'password': password, 'email': email, 'project_name': project, 'api_version': 'v1', } } app_svc = UserGroupSvc(bll_request=BllRequest(bll_request)) reply = app_svc.handle() return reply['data'] def _get_user_list(self): bll_request = { api.AUTH_TOKEN: self.token, 'target': 'user_group', 'data': { 'operation': 'users_list', 'api_version': 'v1', } } app_svc = UserGroupSvc(bll_request=BllRequest(bll_request)) reply = app_svc.handle() return reply['data'] def _del_user(self, user_id, project_id=None): scoped = False if project_id is not None: scoped = True users = [user_id] bll_request = { api.AUTH_TOKEN: self.token, 'target': 'user_group', 'data': { 'operation': 'users_remove', 'user_ids': users, 'api_version': 'v1', } } if scoped is True: bll_request['data']['project_id'] = project_id app_svc = UserGroupSvc(bll_request=BllRequest(bll_request)) app_svc.handle() def _user_cleanup(self): users = self._get_user_list() for user in users: if user['username'].startswith('test_'): user_id = user['user_id'] project_id = user['project_id'] self._del_user(user_id, project_id) def test_default_project(self): bll_request = { api.AUTH_TOKEN: self.token, 'target': 'user_group', 'data': { 'operation': 'unused' } } app_svc = UserGroupSvc(bll_request=BllRequest(bll_request)) svc = get_service_tenant_name() # Tests a a bunch of tuples with input, and expect output as items tests = [ (["admin", svc], None), (["admin"], None), ([svc], None), (["admin", svc, "demo"], "demo"), (["demo"], "demo"), (["demo", "foo"], "demo"), (["admin", svc, "foo"], "foo"), (["admin", svc, "demo"], "demo"), (["foo"], "foo"), ] # Test all of the above scenarios for vals, expected in tests: self.assertEquals(expected, app_svc._select_default_project(vals), ",".join(vals)) def newuser(self): return "test_%s" % randomidentifier() 070701000A47E6000081A40000000000000000000000015B64B4DF00002740000000FD0000000200000000000000000000004200000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/tests/util.py# (c) Copyright 2015-2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017-2018 SUSE LLC from contextlib import contextmanager from bll import api from bll.api import auth_token from bll.api.auth_token import verify from bll.api.request import BllRequest from bll.common import job_status from keystoneclient.v3 import client as ksclient3 from bll.plugins.service import SvcBase from stubs.common.dict_job_status import DictStatus from keystoneclient.v2_0 import client as ksc import logging import os import mock import pymysql.cursors import random import socket import unittest from pecan import testing import pecan detected = None def functional(requires=None): """ This function is used as a decorator to indicate that the given tests are "functional", which is to say, that they generally depend on some additional services to be running in the environment which are not automatically started by the test. :param requires: comma-separated list of items that must appear in the environment variable named "functional". For legacy reasons, if "functional" consists solely of the string "1", "true", or "all", then it will be considered that all requirements are met """ missing = _functional_lacking(requires) return unittest.skipUnless(len(missing) == 0, "Functional test requires %s" % (",".join(missing))) def _functional_lacking(requires=None): """ Returns a set of items that are lacking in order to execute a functional test that requires the indicated items. Returns an empty set if all necessary items are present. """ functional_env = os.environ.get('functional', '') needs = {x.strip() for x in requires.split(",")} if functional_env in ('all', 'true', '1'): has = needs else: has = {x.strip() for x in functional_env.split(",")} global detected if 'auto' in has: if detected is None: detected = set() if _has_mysql(): detected.add('mysql') services = [] try: auth_ref = get_auth_ref_from_env() services = ksc.Client(auth_ref=auth_ref, verify=verify()).services.list() except: pass # Add both the name and type as being detected, in case # some tests indicate the need for the name (e.g. monasca) # while others indicate the need for the type (e.g. volume) for s in services: detected.add(s.type) detected.add(s.name) print("Autodetected:", ",".join(sorted(detected))) has |= detected missing = needs.difference(has) return missing def _has_mysql(): # Retrieve a database connection based on the config try: loadConfigFile() config = pecan.conf.db.to_dict() config['cursorclass'] = pymysql.cursors.DictCursor pymysql.connect(**config) return True except: return False def _is_port_in_use(address, port): s = socket.socket() try: s.connect((address, port)) return True except socket.error: return False def getConfigFile(): test_config = os.path.realpath( os.path.join(os.path.dirname(__file__), '../tests/config.py')) if os.getenv("BLL_CONF_OVERRIDE"): config_file = os.getenv("BLL_CONF_OVERRIDE") elif os.path.isfile(test_config): config_file = test_config elif os.path.isfile('/etc/opsconsole-server/config.py'): config_file = '/etc/opsconsole-server/config.py' else: config_file = '/etc/opsconsole-server/opsconsole-server.conf' return config_file loaded_config = False def loadConfigFile(): global loaded_config if loaded_config: return loaded_config = True pecan.conf.update( pecan.configuration.conf_from_file(getConfigFile())) # Use the logging configuration in the config file if 'logging' in pecan.conf: logging.config.dictConfig(pecan.conf['logging']) else: logging.basicConfig(level=logging.WARN) class TestCase(unittest.TestCase): @classmethod def load_test_app(cls): cls.app = testing.load_test_app(getConfigFile()) @classmethod def setUpClass(cls): loadConfigFile() # Determine whether the job status functions should run with mysql # or a test double (DictStatus). It is basically equivalent to # determining whether 'mysql' is in the functional environment # variable, except that it also takes into account the new 'auto' # option. if _functional_lacking('mysql'): # Inject a mock, since mysql is not available job_status._get_status_obj = mock.Mock(return_value=DictStatus()) def randomword(length=12): """ Generate a string of random alphanumeric characters """ valid_chars = \ 'abcdefghijklmnopqrstuvyxwzABCDEFGHIJKLMNOPQRSTUVYXWZ012345689' return ''.join(random.choice(valid_chars) for i in range(length)) def randomidentifier(length=12): """ Generate a random identifier, which is an alpha character followed by alphanumeric characters """ valid_firsts = 'abcdefghijklmnopqrstuvyxwzABCDEFGHIJKLMNOPQRSTUVYXWZ' return ''.join((random.choice(valid_firsts), randomword(length - 1))) def randomhex(length=32): """ Generate a random hex string """ valid_chars = 'abcdef0123456789' return ''.join(random.choice(valid_chars) for i in range(length)) def randomurl(): """ Generate a random url value """ return "http://%s/%s" % (randomword(), randomword()) def randomdict(): # build a complicated, nested dictionary my_dict = {} my_dict[randomidentifier()] = randomword() nested_dict = {} nested_dict[randomidentifier()] = randomword() my_dict['dict'] = nested_dict nested_array = [] nested_array.append(randomword()) nested_array.append(randomword(500)) nested_array.append(random.random()) nested_array.append(random.randint(0, 1000)) my_dict['array'] = nested_array return my_dict def randomip(): return '%d.%d.%d.%d' % (random.randint(1, 255), random.randint(0, 255), random.randint(0, 255), random.randint(0, 255)) def test_spawn_service(target, operation=None, action=None, token=None, data=None): """ Variant of spawn_service intended for testing -- the keystone token will be faked if not passed """ payload = {} if operation: payload[api.OPERATION] = operation if data: payload.update(data) req = { api.TARGET: target, api.ACTION: action or "GET", api.AUTH_TOKEN: token or get_mock_token(), api.DATA: payload } return SvcBase.spawn_service(BllRequest(req)) def get_mock_token(): return randomhex() @contextmanager def log_level(level, name): """ Enable modifying logging level in a context manager """ logger = logging.getLogger(name) old_level = logger.getEffectiveLevel() logger.setLevel(level) try: yield logger finally: logger.setLevel(old_level) def get_auth_ref_from_env(): """ Obtains keystone token using the provided configuration """ username = os.getenv('OS_USERNAME') password = os.getenv('OS_PASSWORD') if username is None or password is None: raise Exception("OS_USERNAME and OS_PASSWORD env vars must be set") return auth_token.login(username, password) def get_token_from_env(): return get_auth_ref_from_env().auth_token ksclient = None def create_user(project_roles={}, domain_roles={}): # Creates a keystone user with the specified roles. Project_roles # and domain_roles are dictionaries with the key(s) being the name of the # project (or domain), and the value(s) being the name of the role # global ksclient if ksclient is None: # The OS_USERNAME and OS_PASSWORD should be for a keystone admin, # a user with the admin role in the default domain, since it is being # used to create new users. # Need a v3 URL for performing domain operations ksclient = ksclient3.Client(auth_url=auth_token.get_auth_url('v3'), username=os.getenv('OS_USERNAME'), password=os.getenv('OS_PASSWORD'), domain_name='default', verify=verify()) username = "test_" + randomidentifier() password = randomidentifier() if not project_roles or 'admin' in project_roles: project_name = 'admin' else: # Just pick a project project_name = project_roles.keys()[0] domains = {d.name: d for d in ksclient.domains.list()} projects = {p.name: p for p in ksclient.projects.list()} roles = {r.name: r for r in ksclient.roles.list()} user = ksclient.users.create(name=username, password=password, default_project=projects[project_name]) for project, role in project_roles.iteritems(): ksclient.roles.grant(role=roles[role], user=user, project=projects[project]) for domain, role in domain_roles.iteritems(): ksclient.roles.grant(role=roles[role], user=user, domain=domains[domain]) # Return the new user and generated password return user, password def delete_user(user): global ksclient if ksclient is not None and user is not None: ksclient.users.delete(user) 07070100080FF3000041ED0000000000000000000000025B64B4DF00000000000000FD0000000200000000000000000000003A00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/tools07070100080FFC000081A40000000000000000000000015B64B4DF00000A2B000000FD0000000200000000000000000000004A00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/tools/install_venv.py# (c) Copyright 2015-2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC # Copyright 2010 United States Government as represented by the # Administrator of the National Aeronautics and Space Administration. # All Rights Reserved. # # Copyright 2010 OpenStack Foundation # Copyright 2013 IBM Corp. # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. import ConfigParser import os import sys import install_venv_common as install_venv # flake8: noqa def print_help(project, venv, root): help = """ %(project)s development environment setup is complete. %(project)s development uses virtualenv to track and manage Python dependencies while in development and testing. To activate the %(project)s virtualenv for the extent of your current shell session you can run: $ source %(venv)s/bin/activate Or, if you prefer, you can run commands in the virtualenv on a case by case basis by running: $ %(root)s/tools/with_venv.sh """ print help % dict(project=project, venv=venv, root=root) def main(argv): root = os.path.dirname(os.path.dirname(os.path.realpath(__file__))) if os.environ.get('tools_path'): root = os.environ['tools_path'] venv = os.path.join(root, '.venv') if os.environ.get('venv'): venv = os.environ['venv'] pip_requires = os.path.join(root, 'requirements.txt') test_requires = os.path.join(root, 'test-requirements.txt') py_version = "python%s.%s" % (sys.version_info[0], sys.version_info[1]) setup_cfg = ConfigParser.ConfigParser() setup_cfg.read('setup.cfg') project = setup_cfg.get('metadata', 'name') install = install_venv.InstallVenv( root, venv, pip_requires, test_requires, py_version, project) options = install.parse_args(argv) install.check_python_version() install.check_dependencies() install.create_virtualenv(no_site_packages=options.no_site_packages) install.install_dependencies() install.post_process() print_help(project, venv, root) if __name__ == '__main__': main(sys.argv) 07070100080FF5000081A40000000000000000000000015B64B4DF00001EAE000000FD0000000200000000000000000000005100000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/tools/install_venv_common.py# (c) Copyright 2015-2016 Hewlett Packard Enterprise Development LP # (c) Copyright 2017 SUSE LLC # Copyright 2013 OpenStack Foundation # Copyright 2013 IBM Corp. # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. """Provides methods needed by installation script for OpenStack development virtual environments. Since this script is used to bootstrap a virtualenv from the system's Python environment, it should be kept strictly compatible with Python 2.6. Synced in from openstack-common """ from __future__ import print_function import optparse import os import subprocess import sys class InstallVenv(object): def __init__(self, root, venv, requirements, test_requirements, py_version, project): self.root = root self.venv = venv self.requirements = requirements self.test_requirements = test_requirements self.py_version = py_version self.project = project def die(self, message, *args): print(message % args, file=sys.stderr) sys.exit(1) def check_python_version(self): if sys.version_info < (2, 7): self.die("Need Python Version >= 2.7") def run_command_with_code(self, cmd, redirect_output=True, check_exit_code=True): """Runs a command in an out-of-process shell. Returns the output of that command. Working directory is self.root. """ if redirect_output: stdout = subprocess.PIPE else: stdout = None proc = subprocess.Popen(cmd, cwd=self.root, stdout=stdout) output = proc.communicate()[0] if check_exit_code and proc.returncode != 0: self.die('Command "%s" failed.\n%s', ' '.join(cmd), output) return (output, proc.returncode) def run_command(self, cmd, redirect_output=True, check_exit_code=True): return self.run_command_with_code(cmd, redirect_output, check_exit_code)[0] def get_distro(self): if (os.path.exists('/etc/fedora-release') or os.path.exists('/etc/redhat-release')): return Fedora( self.root, self.venv, self.requirements, self.test_requirements, self.py_version, self.project) else: return Distro( self.root, self.venv, self.requirements, self.test_requirements, self.py_version, self.project) def check_dependencies(self): self.get_distro().install_virtualenv() def create_virtualenv(self, no_site_packages=True): """Creates the virtual environment and installs PIP. Creates the virtual environment and installs PIP only into the virtual environment. """ if not os.path.isdir(self.venv): print('Creating venv...', end=' ') if no_site_packages: self.run_command(['virtualenv', '-q', '--no-site-packages', '-p', '/usr/bin/python', self.venv]) else: self.run_command(['virtualenv', '-q', '-p', '/usr/bin/python', self.venv]) print('done.') else: print("venv already exists...") pass def pip_install(self, *args): self.run_command(['tools/with_venv.sh', 'pip', 'install', '--upgrade'] + list(args), redirect_output=False) def install_dependencies(self): print('Installing dependencies with pip (this can take a while)...') # First things first, make sure our venv has the latest pip and # setuptools and pbr self.pip_install('pip>=1.4') self.pip_install('setuptools') self.pip_install('pbr') self.pip_install('-r', self.requirements, '-r', self.test_requirements) def post_process(self): self.get_distro().post_process() def parse_args(self, argv): """Parses command-line arguments.""" parser = optparse.OptionParser() parser.add_option('-n', '--no-site-packages', action='store_true', help="Do not inherit packages from global Python " "install") parser.add_option('-v', '--venv', action='store', help="path for virtual environment", dest="venv") options = parser.parse_args(argv) if options[0].venv is not None: self.venv = options[0].venv return options[0] class Distro(InstallVenv): def check_cmd(self, cmd): return bool(self.run_command(['which', cmd], check_exit_code=False).strip()) def install_virtualenv(self): if self.check_cmd('virtualenv'): return if self.check_cmd('easy_install'): print('Installing virtualenv via easy_install...', end=' ') if self.run_command(['easy_install', 'virtualenv']): print('Succeeded') return else: print('Failed') self.die('ERROR: virtualenv not found.\n\n%s development' ' requires virtualenv, please install it using your' ' favorite package management tool' % self.project) def post_process(self): """Any distribution-specific post-processing gets done here. In particular, this is useful for applying patches to code inside the venv. """ pass class Fedora(Distro): """This covers all Fedora-based distributions. Includes: Fedora, RHEL, CentOS, Scientific Linux """ def check_pkg(self, pkg): return self.run_command_with_code(['rpm', '-q', pkg], check_exit_code=False)[1] == 0 def apply_patch(self, originalfile, patchfile): self.run_command(['patch', '-N', originalfile, patchfile], check_exit_code=False) def install_virtualenv(self): if self.check_cmd('virtualenv'): return if not self.check_pkg('python-virtualenv'): self.die("Please install 'python-virtualenv'.") super(Fedora, self).install_virtualenv() def post_process(self): """Workaround for a bug in eventlet. This currently affects RHEL6.1, but the fix can safely be applied to all RHEL and Fedora distributions. This can be removed when the fix is applied upstream. Nova: https://bugs.launchpad.net/nova/+bug/884915 Upstream: https://bitbucket.org/eventlet/eventlet/issue/89 RHEL: https://bugzilla.redhat.com/958868 """ if os.path.exists('contrib/redhat-eventlet.patch'): # Install "patch" program if it's not there if not self.check_pkg('patch'): self.die("Please install 'patch'.") # Apply the eventlet patch self.apply_patch(os.path.join(self.venv, 'lib', self.py_version, 'site-packages', 'eventlet/green/subprocess.py'), 'contrib/redhat-eventlet.patch') 07070100080FFB000081ED0000000000000000000000015B64B4DF00000054000000FD0000000200000000000000000000004700000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/tools/with_venv.sh#!/bin/bash TOOLS=`dirname $0` VENV=$TOOLS/../.venv source $VENV/bin/activate && $@ 07070100081055000081A40000000000000000000000015B64B4DF00000327000000FD0000000200000000000000000000003C00000000ardana-opsconsole-server-8.0+git.1533326559.d98c230/tox.ini[tox] minversion = 1.6 envlist = py33,py34,py26,py27,pypy,pep8 skipsdist = True [testenv] usedevelop = True install_command = pip install {opts} {packages} setenv = VIRTUAL_ENV={envdir} deps = -r{toxinidir}/requirements.txt -r{toxinidir}/test-requirements.txt commands = /bin/bash run_tests.sh -N {posargs} [testenv:pep8] commands = /bin/bash run_tests.sh -N --pep8 [testenv:venv] commands = {posargs} [testenv:cover] commands = python setup.py testr --coverage --testr-args='{posargs}' [testenv:docs] commands = python setup.py build_sphinx [flake8] # H803 skipped on purpose per list discussion. # E123, E125 skipped as they are invalid PEP-8. show-source = True ignore = E123,E125,H803 builtins = _ exclude=.venv,.git,.tox,dist,doc,*openstack/common*,*lib/python*,*egg,build,.ropeproject 07070100000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000B00000000TRAILER!!!926 blocks