enhancement on DB operations in import #142

immarvin · 2018-10-25T10:10:48Z

Description:

fix the non-group objects import issue Some orphan entries are exported as "groups" and then created as nodegroups when imported. #85 , the multiple key table related issues the export of switch table entries inxcat-inventory #140 and xcat-inventory import support multiple keys object passwd #138, the import -c support for site object issue "xcat-inventory import -c -t site -f xxx" should clear the site table first #139
support the mapping from multiple key table to individual objects or subset attributes of an object
support the reference of sub inventory objects in object definition yaml with REF{<sub inventory object>} syntax
the inventory schema grows to version 2.0, a sub inventory type networkconn is added to describe the network connection info in switch table
UT:

switch table is mapped to network_info. connections of node object , export node object:

[root@c910f03c05k21 inventory]# xcat-inventory export -t node -o testnodes
node:
  testnodes:
    device_info:
      arch: armel
      characteristics: switch
      switchtype: onie
    device_type: switch
    engines:
      hardware_mgt_engine:
        engine_type: switch
      netboot_engine:
        engine_info:
          osimage: cumulus3.5.2-armel
        engine_type: onie
    network_info:
      connections:
      - switch: mid05tor11
        switchport: '22'
      - switch: mid05tor13
        switchport: '22'
      - switch: mid05tor14
        switchport: '22'
      primarynic:
        ip: 10.3.5.141
        mac:
        - 8c:ea:2b:12:ca:40
    obj_info:
      description: Cumulus Linux 3.4.1 (Linux Kernel 4.1.33-1+cl3u9)
      groups: switch,edge_switch
    obj_type: node
    role: compute
    role_info:
      tftpserver: briggs01
schema_version: '2.0'

#Version 2.14.5 (git commit 5bab3f8bac3374cd9cb2e26b4a71c7cb32bb1126, built Mon Oct 22 06:16:06 EDT 2018)
[root@c910f03c05k21 inventory]#

import node object:

[root@c910f03c05k21 inventory]# xcat-inventory import -f /tmp/testnodes
loading inventory date in "/tmp/testnodes"
start to import "node" type objects
 preprocessing "node" type objects
 writting "node" type objects
start to import "networkconn" type objects
 preprocessing "networkconn" type objects
 writting "networkconn" type objects
Inventory import successfully!
[root@c910f03c05k21 inventory]# xcat-inventory export -t node -o testnodes
node:
  testnodes:
    device_info:
      arch: armel
      characteristics: switch
      switchtype: onie
    device_type: switch
    engines:
      hardware_mgt_engine:
        engine_type: switch
      netboot_engine:
        engine_info:
          osimage: cumulus3.5.2-armel
        engine_type: onie
    network_info:
      connections:
      - switch: mid05tor11
        switchport: '22'
      - switch: mid05tor13
        switchport: '22'
      - switch: mid05tor14
        switchport: '24'
      primarynic:
        ip: 10.3.5.141
        mac:
        - 8c:ea:2b:12:ca:40
    obj_info:
      description: Cumulus Linux 3.4.1 (Linux Kernel 4.1.33-1+cl3u9)
      groups: switch,edge_switch
    obj_type: node
    role: compute
    role_info:
      tftpserver: briggs01
schema_version: '2.0'

export of passwd:

[root@c910f03c05k21 inventory]# xcat-inventory export -t passwd
passwd:
  ipmi:
    password: '''USERID'''
    username: '''admin'''
  openbmc:
  - password: xcat
    username: mmmmmmmm
  - password: USERID
    username: ssss
  switch:
    password: admin
    username: root
  system:
  - password: cluster
    username: root
  - password: cluster
    username: wsuser
schema_version: '2.0'

import of passwd:

[root@c910f03c05k21 inventory]# cat /tmp/passwd
passwd:
  ipmi:
    password: '''USERID'''
    username: '''admin'''
  openbmc:
  - password: 'USERID'
    username: 'ssss'
  switch:
    password: admin
    username: root
  system:
  - password: cluster
    username: root
  - password: cluster
    username: wsuser
schema_version: '2.0'

#Version 2.14.5 (git commit 5bab3f8bac3374cd9cb2e26b4a71c7cb32bb1126, built Mon Oct 22 06:16:06 EDT 2018)
[root@c910f03c05k21 inventory]# xcat-inventory import -f /tmp/passwd -c
loading inventory date in "/tmp/passwd"
start to import "passwd" type objects
 preprocessing "passwd" type objects
 writting "passwd" type objects
Inventory import successfully!
[root@c910f03c05k21 inventory]# tabdump passwd
#key,username,password,cryptmethod,authdomain,comments,disable
"ipmi","'admin'","'USERID'",,,,
"openbmc","ssss","USERID",,,,
"switch","root","admin",,,,
"system","root","cluster",,,,
"system","wsuser","cluster",,,,
[root@c910f03c05k21 inventory]#

… file still left afer import

bybai · 2018-11-01T03:18:05Z

xcat-inventory/xcclient/inventory/schema/2.0/1

@@ -0,0 +1,280 @@
+!!python/dict


xcat-inventory/xcclient/inventory/schema/2.0/1 is not valid schema file name?

bybai · 2018-11-01T03:27:10Z

xcat-inventory/xcclient/inventory/dbfactory.py

@@ -244,23 +331,27 @@ def cleartab(self,tabs):
                flattabs.append(tab)
            else:
                matrixtabs.append(tab)
-        for tab in matrixtabs:
+        for tab in tabs:
            if hasattr(dbobject,tab):
                tabcls=getattr(dbobject,tab)


line 323~333: to generate flattabs and matrixtabs. But these 2 lists are not used by function cleartab

bybai · 2018-11-01T05:48:59Z

xcat-inventory/xcclient/inventory/dbfactory.py


-        if  tabkey != item and newdict[item] is None:
+        if  item not in objkeys and newdict[item] is None:
            newdict[item]=''

        #delete table rows when (1)the key is None or blank (2)the key is not specified in newdict and all non-key values are blank 


could you update the comments according to new codes?

…ing import

bybai · 2018-11-01T06:48:41Z

xcat-inventory/xcclient/inventory/dbfactory.py

        #print(tabdict)
        #print("\n")
        if tabdict is None:
            return None
        for key in tabdict.keys():
            utils.verbose("  writting object: "+str(key),file=sys.stdout)
+            #clear any existing table entries before adding new entries
+            df=dbfactory(self._dbsession) 
+            df.cleartab(tabdict[key].keys(),[key])


When it is not update, manager.py calls cleartab to clear up db, there is no data in DB now. In line 171 call cleartab, it clear up DB again in for loop. And if there are too many nodes, for each node, it clear up switch table once, the performance can be enhanced here.

will refine this in following sprints

I opened a ticket to trace this enhance #147, thanks

bybai · 2018-11-01T07:55:25Z

I agree to merge this pull request for functions fixing , the DB execution performance enhancement ticket (#147) can be planned in another sprint. Thanks

immarvin added 3 commits October 19, 2018 06:24

support multiple key in table

f04315c

refine logic

d0428e7

multiple key table export

9f6dfa2

immarvin changed the title ~~Ondb~~ multiple key table support Oct 26, 2018

import support

84b5bfc

immarvin changed the title ~~multiple key table support~~ multiple key table support[DO NOT MERGE] Oct 26, 2018

immarvin mentioned this pull request Oct 26, 2018

the export of switch table entries inxcat-inventory #140

Closed

refine code logic

3c739aa

immarvin changed the title ~~multiple key table support[DO NOT MERGE]~~ multiple key table support Oct 31, 2018

immarvin force-pushed the ondb branch from e4385fa to c0c1d14 Compare October 31, 2018 09:21

fix issues that existing entries in DB but not specified in inventory…

8a3afe5

… file still left afer import

immarvin force-pushed the ondb branch from c0c1d14 to 8a3afe5 Compare November 1, 2018 02:44

This was referenced Nov 1, 2018

Some orphan entries are exported as "groups" and then created as nodegroups when imported. #85

Closed

"xcat-inventory import -c -t site -f xxx" should clear the site table first #139

Closed

immarvin changed the title ~~multiple key table support~~ enhancement on DB operations in import Nov 1, 2018

This was referenced Nov 1, 2018

xcat-inventory import support multiple keys object passwd #138

Closed

xcat-inventory support multiple object passwd enhancement #135

Closed

immarvin requested a review from bybai November 1, 2018 03:03

immarvin added this to the 2.14.5 milestone Nov 1, 2018

immarvin assigned bybai Nov 1, 2018

immarvin added bug Something isn't working enhancement New feature or request labels Nov 1, 2018

bybai reviewed Nov 1, 2018

View reviewed changes

refine schema directory hierarchy

f03a1f7

immarvin force-pushed the ondb branch from 3b33fc2 to f03a1f7 Compare November 1, 2018 05:58

immarvin added 2 commits November 1, 2018 02:41

remove referenced sub inventory objects from whole inventory dict dur…

5cb8183

…ing import

remove unused lines

686e893

bybai reviewed Nov 1, 2018

View reviewed changes

refine comments

4bf9969

bybai merged commit 8bfa208 into xcat2:master Nov 1, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

enhancement on DB operations in import #142

enhancement on DB operations in import #142

immarvin commented Oct 25, 2018 •

edited

Loading

bybai Nov 1, 2018 •

edited

Loading

immarvin Nov 1, 2018

bybai Nov 1, 2018

immarvin Nov 1, 2018

bybai Nov 1, 2018

immarvin Nov 1, 2018

bybai Nov 1, 2018

immarvin Nov 1, 2018

bybai Nov 1, 2018

bybai commented Nov 1, 2018

enhancement on DB operations in import #142

enhancement on DB operations in import #142

Conversation

immarvin commented Oct 25, 2018 • edited Loading

bybai Nov 1, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bybai commented Nov 1, 2018

immarvin commented Oct 25, 2018 •

edited

Loading

bybai Nov 1, 2018 •

edited

Loading