This document lists gaps between the current set of information stored in OSG/OIM, WLCG/GOCDB and information needed to auto generate Meshconfigs.
Goal of this project is to take information sources such as these..
I've given pointer for both AGIS and GOCDB to load data for WLCG. I am not sure which ones I need to be using..
- http://atlas-agis-api.cern.ch/request/service/query/get_services/?json&type=PerfSonar&status=ANY (for mesh config)
- http://atlas-agis-api.cern.ch/request/site/query/list_rcsites/?json&state=ANY (for list of sites and their country)
(list of other AGIS APIs http://atlas-agis-api.cern.ch/request/ and some info on REST itself http://atlas-agis.cern.ch/docs/latest/restfullapi/atlassitesmatrix.html?highlight=link)
And use that information to generate mesh configs that looks like following
WLCG (Currently updated manually - so I heard)
- (For test definitions) https://grid-deployment.web.cern.ch/grid-deployment/wlcg-ops/perfsonar/conf/central/testdefs/jsons/
- (For site definitions) https://grid-deployment.web.cern.ch/grid-deployment/wlcg-ops/perfsonar/conf/central/sitedefs/jsons/
OSG (Already auto-generated but it's still a prototype)
Both WLCG / OSG information sources are missing some information needed to produce above mesh configs, and goal of this document is to list all such "gaps" and propose on how to store & expose such information.
Item 1. Table of published mesh configs and "tests" under each mesh config (& resource membership)
We need to define a list of mesh configs to be generated (for each "area"?) and for each mesh config define a list of "tests" which contains the test parameter and test type ("mesh", "disjoint", or "star"). disjoint and star tests can have 2 member lists (A list and B list) (mesh has only 1)
"meshconfigs" table should be created to host a list of mesh config with following attributes.
"membergroups" table will defines a list of membergroups (but not the actual hosts) with following attributes.
"meshconfigs-tests" table will host a list of "tests" for each mesh config (1-to-many) with following attributes.
(owamp test sample)
(bwctl test sample)
OIM will allow resource services to hold "service attributes" key-value pairs (similar to GOCDB's service extension properties?). For perfsonar toollkit service, we will add a new key; "MESHCONFIG_MEMBER_GROUP_IDS" with values set to delimited IDs from "membergroups" table above. Resource administrator for each perfsonar toolkit instance can edit the list of membergroup IDs (using the actual names instead of IDs)
** I will filter membergroups by either bandwidth or latency stored in membergroup table.. We shouldn't be allowing Bandwidth service instance to be member of Latency group as shown in the screenshot..
User interfaces to allow OSG support staff to define meshconfig / membergroups information will need to be implemented. (Add / Remove /Edit etc..). The list of mesh/tests themselves should be published via MyOSG apart from the actual mesh configs generated.
Same as OSG - unless this information is already available at URL I am not aware of.
Some of these information can be stored as service extension properties (https://wiki.egi.eu/wiki/GOCDB/Input_System_User_Documentation#Extension_Properties) and exposed under /query/get_services API but it's up to GOCDB
Item 2. Organization and Site administrators
Organization administrator is currently missing, but OIM can provide site and host administrator from OIM support center and resource admin information. I believe OIM has enough contact information to generate a useful mesh config.
I see email address available under site, but not full name. Organization and host administrator information is missing, but I am not sure if they are needed. If site site admin info is enough, then AGIS just have to provide admin contact name.
Following is a sample of site information AGIS. Please notice that only email address is available.
(absorbed into Item 1)
Item 4. Primary and Secondary perfsonar toolkit endpoints for each CE and SE
We need to be able to specify which perfsonar toolkit instance should be used to analyze information for each SE/CE (I think we should have some kind of public service that allows user to enter an IP/host and tell user nearest perfsonar instance.. but that's another story).
Under OIM/Resource/CE&SE services, we will add a select box allowing administrator to enter primary and secondary perfsonar toolkit instance responsible for monitoring that CE or SE.
The information will be exposed as part of resource service information via MyOSG.
Similar to OIM. The information should be stored as service extension properties and exposed under /query/get_services API
OSG mesh (http://myosg.grid.iu.edu/miscpfmesh?count_sg_1&count_active=on&count_enabled=on) are currently generated using following mapping rules.
- VO ownership information is used to create different mesh configs.
- oim/facility == organizations (description) *org admin information is currently missing.
- oim/site == site (location, description)
- oim/support center admin == site admin *not sure if this is the right thing to do.
- oim/resource/service == host
- oim/resource/admin == host_administrator
Mesh config structure
The "green" text shows optional information and the "red" text shows required (at least by WLCG policy).