The influence of community recommendations on metadata completeness

Many communities use standard, structured documentation that is machine-readable, i.e. metadata, to make discovery, access, use, and understanding of scientific datasets possible. Organizations and communities have also developed recommendations for metadata content that is required or suggested for their data developers and users. These recommendations are typically specific to metadata representations (dialects) used by the community. By considering the conceptual content of the recommendations, quantitative analysis and comparison of the completeness of multiple metadata dialects becomes possible. This is a study of completeness of EML and CSDGM metadata records from DataONE in terms of the LTER recommendation for Completeness. The goal of the study is to quantitatively measure completeness of metadata records and to determine if metadata developed by LTER is more complete with respect to the recommendation than other collections in EML and in CSDGM. We conclude that the LTER records are broadly more complete than the other EML collections, but similar in completeness to the CSDGM collections.

• Metadata recommendations as a community activity to improve completeness • Quantitative measures of recommendation completeness • Comparison of EML and CSDGM usage across DataONE using a conceptual version of the LTER Recommendation for Completeness • EML metadata created by the LTER data managers is broadly more complete than EML metadata created by other DataONE member nodes. • LTER in the middle of the pack when compared to member nodes using CSDGM

Abstract
Many communities use standard, structured documentation that is machinereadable, i.e. metadata, to make discovery, access, use, and understanding of scientific datasets possible. Organizations and communities have also developed recommendations for metadata content that is required or suggested for their data developers and users. These recommendations are typically specific to metadata representations (dialects) used by the community. By considering the conceptual content of the recommendations, quantitative analysis and comparison of the completeness of multiple metadata dialects becomes possible. This is a study of completeness of EML and CSDGM metadata records from DataONE in terms of the LTER recommendation for Completeness. The goal of the study is to quantitatively measure completeness of metadata records and to determine if metadata developed by LTER is more complete with respect to the recommendation than other collections in EML and in CSDGM. We conclude that the LTER records are broadly more complete than the other EML collections, but similar in completeness to the CSDGM collections.

Introduction
Scientists and scientific communities recognize the need to document observations and processing clearly and completely to support discovery, access, use, understanding and reproducibility of their scientific results. Many datasets and products are documented using approaches and tools developed by data collectors to support their own analysis and understanding needs. This documentation can exist in almost any conceivable form, each with associated storage and preservation strategies. This custom, often unstructured, approach may work well for independent investigators or in the confines of a laboratory or community, but it makes it difficult for users outside of these small groups to discover, use, and understand the data without consulting with its creators.
Metadata, in contrast to documentation, provides well-defined content in structured representations that make it easier to share and discover. This makes it possible for users to access and quickly understand many aspects of datasets that they have not collected or created themselves but need to answer specific questions. It also makes it possible to integrate information into discovery and analysis tools, and to provide consistent references from the metadata to external documentation.

Metadata Standards, Concepts, Dialects, and Recommendations
Scientific communities that recognize the need for metadata typically address that need using one of several approaches: they either use a metadata standard proposed by a related community or organization, or they develop a community standard. In most cases, they also include a standard representation for the metadata. We refer to these representations as metadata dialects. These metadata dialects include concept names, definitions and associated structures. A concept is a general, dialect-independent term for describing a documentation entity, typically an element or attribute defined in XML. Typically, the communities or organizations that develop the standard also develop a set of recommendations for metadata content. We refer to these as metadata recommendations.
The relationship between dialects and recommendations is illustrated in Figure 1 using the LTER recommendation that was created for use with the EML dialect, and the FGDC recommendation and the corresponding CSDGM dialect as examples. LTER uses the EML dialect (D1) and their recommendation has five levels: Identification, Discovery, Evaluation, Access, and Integration (R1, R2, R3, R4, R5). All the concepts in the recommendation are included in the dialect. In some cases, the recommended concepts are required by the XML schema used to implement the dialect, illustrated as R6.
When another community, like FGDC, creates a second dialect CSDGM (D2) with recommendations at three levels: Mandatory, Mandatory if Applicable, and Optional (R7, R8, R9), there is typically overlap between the dialects (most often for discovery content) and the recommendations, e.g. R2 and R8 in Figure 1. More in-depth information about the dialects and recommendations follow in the next section.

Dialects and Recommendations at DataONE
The DataONE Data Catalog ("DataONE Data Catalog," n.d.) provides a unique opportunity to explore relationships between metadata recommendations and dialects. It includes collections of metadata records from over 25 different data providing organizations, or member nodes, in at least six different dialects. The most common dialects are EML and CSDGM.

EML was developed by KNB and LTER ("The Long Term Ecological Research
Network | Long-term, broad-scale research to understand our world," n.d.) to address specific needs of the ecological research community. Many ecological research groups in the U.S. and around the world actively use it. The authors were influenced by both CSDGM and ISO metadata standards, so EML shares characteristics with both standards.
CSDGM is commonly known as FGDC because the U.S. Federal Geographic Data Committee developed the standard. It was the standard and dialect required by the U.S. Government for many years (FGDC). It continues to be used and extended by various scientific communities that need to describe their data geospatially.

The LTER Recommendation
As the ecological research community gained experience with EML, it became clear that many metadata records were not complete or consistent enough to serve important community requirements. To address this problem, a group of LTER metadata experts developed a set of recommendations to help guide the creation and improvement of EML metadata records (EML Best Practices for LTER Sites, 2004). The LTER recommendation includes five levels: Identification, Discovery, Evaluation, Access, and Integration, each of which recommends specific elements designed to provide information about the dataset for a specific use case, or need. The descriptions below are directly from the recommendation.
Identification level metadata is the minimum content for adequate data set discovery in a general cataloging system or repository.
Discovery level metadata should provide as much information as possible to support locating datasets by time, taxa, and/or geographic location in addition to basic identification information. Discovery level EML should include the coverage elements of temporalCoverage (when), taxonomicCoverage (what), and geographicCoverage (where) for the dataset as well as the change history in the maintenance element.
Evaluation level metadata should include detailed descriptions of the project, methods, protocols, and intellectual rights in order for a potential user to evaluate the relevance of the data package for their research study or synthesis project.
Access level metadata should provide a user with all the information needed to access and download the data tables, even if the tables' attributes are not thoroughly described. The tags required at this level specify access control and the physical description of the table.
Integration level metadata should support computer-mediated access and processing of data, and therefore requires that all aspects of the data package be fully described.
In this paper, we focus on the LTER recommendation and metadata in two dialects (EML and CSDGM). The concepts included in the LTER Recommendation are listed in Table 1. All of these concepts are included in the EML dialect and four, underlined in Table 1, are required by the EML schema. Twenty-one of these concepts are included in the CSDGM dialect. Four concepts that do not exist in the CSDGM dialect are shown in italics. Ten concepts that are included in the mandatory FGDC recommendation are shown in bold in Table 1. Comparisons of recommendations across communities can provide important insights into similarities and differences between documentation needs. Table 1 indicates significant overlap between the LTER and FGDC recommendations. In this paper, we focus on metadata evaluation rather than recommendation comparisons, so the FGDC Recommendation is not discussed again.
We are interested in situations where documentation needs of different communities and dialects overlap. Figure 1 shows overlaps between D1 and D2 as well as R2 and R8. Such overlap is common in areas with clear common needs, such as data discovery, but can be less common as the metadata becomes more specialized. To identify these overlaps and do cross-dialect comparisons, the recommendations must be described in terms of fundamental documentation concepts that can be identified in multiple dialects.
A second requirement for meaningful cross dialect comparisons is that some concepts occur in both dialects (see discussion of Figure 3 below). Of course, all the LTER recommendations are in the EML dialect, but they may not be included in other dialects, e.g. R1, R3-5 in Figure 1.
The LTER recommendation was well publicized and supported in the LTER community, so we might expect that LTER metadata records are more complete with respect to this recommendation than other metadata collections. We explore the impact of the LTER recommendation in two ways. First, we compare the completeness of the LTER metadata collection in the DataONE metadata repository to collections from other ecological research groups that use the EML dialect. Second, we extend that comparison to metadata collections in DataONE documented in the CSDGM dialect. We accomplish both comparisons through a conceptual abstraction layer that provides a method of crosswalking dialect and recommendation specific XML elements. For example, the concept "Resource Title" is found in both the EML and CSDGM dialects at a specific location in the resource's documentation. By connecting the structural locations, or dialect definitions in multiple dialects, conceptual recommendations can be measured across dialects. The dialect definitions for the LTER recommendation's concepts in EML and CSDGM are listed in Appendix 1.

Method
We are interested in evaluating completeness of metadata collections in multiple dialects with respect to a recommendation made in a single dialect. Our approach is illustrated in Figure 2 which shows two dialects, a conceptual recommendation with two levels (L1 and L2) in Dialect 1, implementations of the recommendation in dialects 1 and 2, and two metadata collections in each dialect.
Typically, recommendations are associated with a native dialect, as illustrated in Figure 1 with R1-5 and D1, so they include an implementation in that dialect. The first step in our analysis is to map those implementations (H-N) to dialect-independent documentation concepts (A-G). For example, the recommendation might recommend that the metadata include an XML element <title> that holds a dataset title and an element <pointOfContact> that holds the name of a point of contact. These two elements could be mapped to the documentation concepts "Resource Title" and "Resource Contact". These mappings are identified by open, bi-directional arrows in Figure 2. Note that all the recommended concepts can be mapped to implementations in the native dialect, as communities do not recommend concepts that do not exist in their implementations. In the LTER case, the recommendations were originally described as documentation concepts, so this step was not necessary.
Once the implementations are known, the metadata evaluation is straightforward. We examine the metadata records to determine which of the concepts they include. We simplify the illustration here by considering only two concepts (A and E). Figure 2 includes two collections in dialect 1. Implementation H of concept A is included in all four of the records in the first collection (indicated by filled arrows) and in two of the three records in collection 2. Implementation L of concept E is included in two of the four records in collection 1 and all three of the records in collection 2. The "concept occurrence %" of concept A in this collection is 100% and of concept E is 50%. Note that elements may be missing from some collections because they don't make sense for that collection even though they are in a recommendation. For example, some DataONE collections may not include biologic observations so the concept "Taxonomic Extent" may not be needed in their metadata. We measure completeness without considering such explanations.
In many cases, we identify groups of metadata records that include, and therefore are missing, the same concepts. Collection 1 includes two such groups. The first two records are missing concept E and the second two records are not missing either H or L. We term these "signature groups" and identify them by the number of concepts that they are missing in each level of the recommendation. The signature of the first group in collection one is "0 1" as these records are missing zero concepts from L1 and one concept from L2. The signature of the second group is "0 0" as they are missing 0 concepts from L2. Note that low numbers are better in these signatures so "0 0" indicates a complete record and the sum of the signature group is the total number of concepts missing from the records in the group.
Another approach to characterizing completeness is to examine the distribution, i.e. mean and standard deviation, of the number of complete records / concept from each collection. In Figure 2 these completeness % are given for each concept / implementation pair. 8 Figure 2. Schematic diagram of methods used in this study.

Data
DataONE includes many member nodes in many dialects. This section describes the data we sampled.

Dialects
DataONE member nodes include metadata records in many dialects (see Table 2). We retrieved data from all DataONE member nodes that included EML or CSDGM dialects.  Table 3 describes the record counts received from the sampling of the DataONE repository during October 2015, as well as the dialect version the documents are written in. The record count for each member node is the total of all the different dialects and dialect versions described in the Dialect Collections and Counts column. The collections are listed by dialect, EML first, and sorted by collection size.

Analysis
Our analysis followed the steps shown schematically in Figure 2. First, we compared the LTER Recommendation to the EML and CSDGM dialects, then we analyzed the metadata collections in each dialect for completeness with respect to the LTER Recommendation.

Comparison of DataONE dialects and the LTER Recommendation
The first step is to define the LTER recommendation conceptually and map the concepts to the dialects for analysis (upper arrows in Figure 2). We used the EML 2.1.1 schema ("Ecological Metadata Language (EML) Specification," n.d.) to identify EML dialect definitions for the recommended concepts. The mappings are described in Appendix 1.
We expect that implementations of all concepts in each level of the LTER Recommendation exist in the EML dialect, but that some may not exist in the CSDGM dialect. We must determine the dialect maximum values in the CSDGM dialect for each level of the LTER Recommendation.  Figure 3 is a utilization of parallel coordinates in two-dimensional space. Parallel coordinates are like a time series, but do not rely on time as an axis. ("Parallel coordinates," 2017) In this example, the concepts in each recommendation level are being counted for the recommendation itself and each dialect if they are contained within the dialect. These are called the recommendation maximum and the dialect maximums respectively. Since the recommendation and dialects are based on not only each level, but all of these levels together, we connect the coordinates for each with a line. The most significant takeaway here is that the gap between the lines showcases the extent that a dialect can meet the recommendation's documentation goals. The Identification Level contains the most concepts (11) while the other levels contain between two and five concepts.
As expected, the EML dialect (shown as a solid orange line in Figure 3) contains every concept in each of these levels. It completely overlays the recommendation, shown here as a dashed blue line. However, the CSDGM dialect, the dashed green line, is missing one concept in each level except for Access. CSDGM records can only be complete with respect to the CSDGM dialect maximum, so a record in the CSDGM dialect cannot contain all the recommended concepts in any of the LTER levels except for the Access level. Comparisons between dialects and recommendations are important as communities make decisions about recommendations that are important to them and dialects that might be used for their metadata. As new recommendations emerge, communities must decide whether to extend legacy dialects or migrate to a new dialect. Several organizations in DataONE have extended CSDGM to include new concepts. For example, Mercury and Biological Data Profile (BDP) are dialects in DataONE that extend CSDGM to contain taxonomic information in the case of BDP, or a resource identifier in Mercury's case. The result of this extension is that the dialect maximum for BDP in the Discovery level of the LTER Recommendation is the same as the number of concepts in the recommendation level.

Metadata Sampling and Cleanup
We sampled up to 250 records from each member node at DataONE (Mecum, 2015). Collections were separated by dialect version and member node. Many of the collections have idiosyncrasies that result in records that are close to standard but have some simple differences. For example, sometimes records will have a namespace prefix added that is not part of the dialect. Since EML uses the same prefix for all versions, sometimes the version needs to be altered in the files so they all match up. We cleaned up these small problems to facilitate analysis across collections and to ensure accurate recognition of XML elements that correspond to recommended concepts.
We included records from all EML versions except the beta versions at KNB which do not share a root with standard EML. The collections were combined into a single directory for each member node. The namespace prefix "eml" was modified to the EML 2.1.1 version in each record written in a previous version. The collections were then treated as though they were EML 2.1.1 as the LTER recommendation had been in use through all the different versions found in the sample set. The resultant collections, record counts, and collection dialects are described in the following table.

Completeness Analysis
After cleaning up the collections, records were analyzed for completeness and reports that detailed the presence or absence of concepts were generated for each record (lower arrows in Figure 2). The reports were concatenated by collection and imported into Excel workbooks to calculate the average occurrence count of each element, as well as collection level average occurrence for a dialect.

Results
We present the results of the completeness analysis using two approaches shown in Figure 2. First, the concept occurrence % are given for each concept and collection, then the number of missing concepts for each recommendation level and collection are compared.

Concept Occurrence Percentages
Concept occurrence tables show the percentage of each collection's records that contain the content for each concept. The tables include rows for each collection and columns for each recommendation concept. The first three rows show totals for all DataONE collections, all EML collections, and all CSDGM collections. The collections are arranged by decreasing size for each dialect (EML above the dark line and CSDGM below). The columns are arranged by decreasing average completeness. Cells contain a color or a percentage with the following meanings: • Green means every record in the collection contains the concept.
• Yellow represents a concept that the dialect contains but is not in any record in the collection. • Red represents a concept that is not included within the collection dialect.
• The percentage is the % of records in the sample set that contain each concept.
The last two columns in each Table shows the overall completeness for each collection numerically and graphically. The bars are colored to indicate collection averages: DataONE and dialect averages (black), LTER (orange), EML (blue), and CSDGM (yellow).

Identification Level
The identification level of the LTER recommendation includes 11 concepts. The entire DataONE collection is 71% complete for this level, the EML collections are 69% complete, and the CSDGM collections are 76% complete. Only one EML collection (ESA) is more complete than LTER. No CSDGM collections are more complete than LTER.
ESA has the most complete collection at 90%. LTER is next at 83%. NMEPSCOR and CLOEBIRD are 82% complete. Only 6 member nodes have less than two thirds completeness for the level. Resource Title and Author/Originator are complete for all collections, regardless of dialect and most collections are 90+% complete for the next four concepts (except Resource Identifier which is not included in CSDGM). Beyond that, the EML collections fall off quickly while the CSDGM collections remain very complete for Publication Date, Resource Distribution, and Metadata Contact. This reflects the fact that Publication Date and Metadata Contact are mandatory concepts in the CSDGM dialect.
There are incomplete concepts in each collection. Each member node has at least one concept from the level that is unused or unusable in the dialect the collection is documented in, except LTER, KNB, and GLEON. The LTER member node collection contains at least one record that includes each concept in the level. Even the CSDGM records have a high occurrence percentage for schema required concepts: Resource Title, Resource Identifier, Author / Originator, and Resource Contact.

Discovery Level
The discovery level of the LTER recommendation includes four concepts. The entire DataONE collection is 58% complete for this level, the EML collections are 57% complete, and the CSDGM collections are 64% complete. Four EML collections (TERN, GOA, ESA, and CLOEBIRD) are more complete than LTER. Two CSDGM collections (EDACGSTORE and SEAD) are more complete than LTER.
Spatial Extent is the only concept included in every collection while Temporal Extent is in all but one collection. Just under half of the collections don't use Taxonomic Extent at all, and every CSDGM record does not contain taxonomic information, as the dialect does not include the concept. Most collections do not have Maintenance information. Except for 3 records from GLEON and one from CLOEBIRD, the 138 records from LTER are the only EML records that include Maintenance information. CSDGM records all contain the Maintenance concept.
Only four collections are more than two thirds complete for the Discovery Level. Two of these collections, CLOEBIRD and TERN, use the EML dialect. EDACGSTORE and SEAD are the CSDGM collections. Table 6. Concept occurrence percentages for Discovery Level. Concepts with * are mandatory in CSDGM. Green means every record contains the concept, yellow means the dialect contains the concept, but no records do, and red represents a concept that is not in the dialect.

Evaluation Level
The evaluation level of the LTER recommendation includes five concepts. The entire DataOnce collection is 54% complete for this level, the EML collections are 55% complete, and the CSDGM collections are 50% complete. Five EML collections (TERN, PISCO, OneShare, GOA, and GLEON) are more complete than LTER. All CSDGM collections are more complete than LTER.
The KUBI collection does not contain any of the concepts in the Evaluation Level. Every other collection includes the Resource Use Constraints concept. The CSDGM dialect does not include a consistent location for Project Description, so no CSDGM records include it. It is of note that five member nodes that use the EML dialect do not include Project Descriptions in their collections and only four collections exist where you can expect to see a project description at least 90% of the time: GLEON, ONEShare, PISCO and TERN. The LTER sample only contains project descriptions in 40 records, or 16% of the sample.
The Evaluation Level is the first level where a member node's collection is missing every concept. KUBI does not use any of the concepts in the Evaluation level. GOA is the most complete member node at 90% complete for the level. No CSDGM documented collection is more than 60% complete. LTER and the EML average are more complete than the CSDGM average.

Access Level
The access level of the LTER recommendation includes two concepts. The entire DataOnce collection is 61% complete for this level, the EML collections are 54% complete, and the CSDGM collections are 81% complete. Five EML collections (SANPARKS, GOA, GLEON, USANPN, and CLOEBIRD) are more complete than LTER. Three CSDGM collections (EDACGSTORE, USGSCSAS, and NMEPSCOR) are more complete than LTER.
The Access level is close to complete for all the collections documented in the CSDGM dialect. Only CDL and most of the SEAD collection are missing the Resource Format concept. LTER is close to complete in documenting constraints on accessing the resource but only 58% of records contain the resource format.
The Access level has two EML collections and three CSDGM collections with 100% completeness.

Integration Level
The integration level of the LTER recommendation includes three concepts. The entire DataONE collection is 27% complete for this level, the EML collections are 19% complete, and the CSDGM collections are 51% complete. Eight EML collections (TFRI, PISCO, SANPARKS, OneShare, GOA, GLEON, USANPN, and CLOEBIRD) are more complete than LTER. Four CSDGM collections (CDL, EDACGSTORE, USGSCSAS, and NMEPSCOR) are more complete than LTER.
In the Integration level, there are two collections that contain every concept: LTER and KNB. Both member nodes helped to create the EML dialect and continue to use it. No other member nodes even use the Attribute Constraints concept. TFRI is the only other EML using member node whose collection contains the Resource Quality Description Concept. All CSDGM collections contain the Resource Quality Description concept, but CSDGM does not document Attribute Constraints. Of the five collections that do not use the Attribute List concept, SEAD is the only member node that uses CSDGM.
The Integration level is the least complete in both dialects. Four collections do not contain any of the concepts. Table 9. Concept occurrence percentages for Integration Level. . Concepts with * are mandatory in CSDGM. Green means every record contains the concept, yellow means the dialect contains the concept, but no records do, and red represents a concept that is not in the dialect.

Comparing Collection Completeness
The data in Tables 5-9 clearly indicates that comparing completeness with respect to a particular recommendation across collections in multiple dialects is a multi-faceted problem. These Tables provide details about what content is included in and missing from these collections. To get a "big picture" comparison of LTER and the other collections, we compared the number of elements/record for all levels of the recommendation using the z-test for a difference between two means (Z-Test).
The results are shown in Table 10 are listed as z-values of a normal distribution. We are not looking for fine distinctions in this case, so we divide the collections into three groups: collections that are more complete (z < -2.0, green), less complete (z > 2.0, red), and similar (-2.0 <= z <= 2.0, white). The number of collections in each group is shown in Table 11. The overall comparison in the last column of Table 10 indicates that the LTER collection is more complete than eleven of the EML collections, similar to two EML collections (GOA and GLEON), and less complete than one collection (CLOEBIRD that includes just one record). This observation provides the simplest answer to our principal question: the LTER collection is generally more complete than other collections in DataONE that use EML.
At the more detailed level, the picture gets more complicated. The identification and discovery levels are similar to the overall level (they make up over ½ of the recommendation). LTER is more complete than ten or more collections in these levels and similar to GLEON, GOA, and ESA. The Access, Evaluation, and Integration levels include five or less concepts. In these levels LTER is closer to the middle of the pack.
The picture is different when LTER is compared to the CSDGM collections. Overall, LTER is in the middle of the pack with the EDACGSTORE collection. USGSCSAS and NMEPSCOR are more complete than LTER; CDL and SEAD are less complete. This difference is also clear in the last three rows of Table 10 that compare LTER with the EML and CSDGM dialect collections. LTER is more complete in all levels than the EML average and less complete in four of five levels when compared to the CSDGM average.

Conclusions and Further Questions
Many communities and organizations have developed metadata dialects and recommendations for content that should be included in metadata for their community. While these recommendations are created within a single community, they provide an opportunity for influence across groups using the native dialect for the recommendations or even across groups that are using other dialects. We used metadata from twenty DataONE member nodes to determine if metadata completeness could be used to discern this influence. Specifically, we measured completeness of collections with respect to LTER recommendations in the native dialect (EML) and in CSDGM.
Our first conclusion is that EML metadata created by the LTER data managers is broadly more complete than EML metadata created by other DataONE member nodes. This suggests that the LTER data management community was influenced in a positive way by the LTER recommendations. The differences are most pronounced for the Identification and Discovery levels of the recommendation (which account for 60% of the recommendation) and not apparent in the other levels (Access, Evaluation, and Integration). Also, the LTER collection is the only one that contains some content for every recommended concept.
We considered the LTER recommendation at the conceptual level (see Appendix 1) so we could map the recommended concepts to XML elements in the Content Standard for Digital Geospatial Metadata (CSDGM) and compare completeness across dialects. The number of collections was smaller (five vs. fifteen), and the LTER collection was in the middle of the group, i.e. there were two more complete collections, one similar, and two less complete collections.
We presented detailed results for all recommended concepts in all the collections we analyzed (Tables 5-9). These results can be used to identify patterns of completeness across collections and to identify areas where improvements are possible. They may also help communities evaluate existing recommendations using empirical evidence of usage. For example, information about Metadata Contact is missing completely from eight of the fifteen EML collections (see Table 5). Does this suggest that this concept is not important to the EML community or indicate that this information should be added to these records?
The same question could apply to other concepts from the Identification Level: Publisher, Publication Date, Contributor Name, or Resource Distribution. The CSDGM collections are 100% complete for Publication Date and Metadata Contact. Is this because of other recommendations influencing the CSDGM records or is it because of differences in the way these communities create and publish data and metadata?
The observations presented in Tables 5-9 also allow us to differentiate collections into two groups. Homogeneous collections are those that include either 100% or 0% of all concepts. Examples of these include TERN, CDL, and KUBI. For example, in Table 5, all TERN and CDL records include eight concepts and no records include three concepts, and all KUBI records include five concepts and no records include six concepts. All the records in the homogeneous collections have the same completeness.
The second group, heterogeneous collections, show varied levels of completeness across the concepts in Tables 5-9. For example, in Table 5, the completeness levels for LTER vary from 18% to 100% and different for most concepts. Most of the collections we examined are heterogeneous.
The differences between these two types of collections could reflect differences in the governance of the collections, differences in recommendations they follow, or heterogeneity in the collections themselves. For example, the LTER "Collection" includes metadata developed by many different data collection sites using many different approaches, so we might not expect homogeneity across the collection. The same would be true for other collections that develop over time and are generally handcurated. The homogeneous collections are unusual and more interesting. Further study is needed to identify how they achieve this homogeneity across fairly large collections.
In addition to recognizing gaps in collections, these evaluations can identify collections (or records) that are very complete with respect to a recommendation. These "shining examples" can be mined for examples and/or stories that can play an important role in guidance or training for metadata curators. The LTER collection is the only one that we analyzed that includes metadata records that are complete for each recommendation level. It is the best source for shining examples.

Questions
We have presented a quantitative approach to evaluating and comparing completeness of metadata collections with respect to recommendations in native and non-native dialects. These techniques are straightforward and provide a framework for collections comparisons in the context of DataONE and in other repositories. The results answer some questions and raise others.
One of the most interesting observations that emerged is that collections in the CSDGM dialect are generally more complete with respect to the LTER recommendations than the collections in the native dialect of the recommendations (EML). This difference is most pronounced in the Discovery and Integration levels (see Table 10), but it is generally consistent across all the levels of the recommendation, i.e. the CSDGM average is more complete than LTER at all levels in Table 11.
The CSDGM collections are influenced by a recommendation made by the U.S. Federal Geographic Data Committee (FGDC) that includes three levels (Mandatory, Mandatory if Applicable, and Optional) (FGDC). Ten concepts from the LTER recommendation are included in the mandatory FGDC Recommendation (marked with * in Tables 5-9). These concepts are generally complete in the CSDGM collections but are also complete at well above 50% average in the EML collections. If these schema required concepts are missing, validators will reject records, which may be why we see these concepts consistently across the CSDGM collections.
What effect does tool selection have on collection completeness? If a tool for metadata creation does not have an option for a concept in a recommendation, the concept is not documentable. Thus, if one were using a tool where the maximum concepts available are a subset of the dialect maximum for the recommendation, it would not be possible to generate a record with a dialect maximum score. Perhaps CSDGM editors might provide a higher 'tool maximum' for the LTER recommendation than Morpho or Metacat, while more modern editors like PASTA are better able to set information managers up for success with regard to documenting the concepts considered important by LTER.
What effect does time have on record completeness? The LTER sample set may all be from 2005. Would new records from succeeding years be more complete? We are in the progress of improving our sampling methods to examine how completeness evolves with time.
Metadata is created at a number of sites in LTER. Each one has their own organizational requirements and development stories. Can these sites be treated as member nodes in a new analysis and show a stronger case for collection evolution towards completeness through community usage of a recommendation by identifying the individual metadata evolution stories at LTER? aspect of a resource. Can be one of several types. /themekey CSDGM /metadata/idinfo/keywords/place/ placekey EML /eml:eml/*/keywordSet/keyword

Resource Distribution
Information about how the resource is available, online, offline, inline.

LTER Discovery
Discovery level metadata should provide as much information as possible to support locating datasets by time, taxa, and/or geographic location in addition to basic identification information. Discovery level EML should include the coverage elements of temporalCoverage (when), taxonomicCoverage (what), and geographicCoverage (where) for the dataset as well as the change history in the maintenance element.

Concept Description Dialect (Fit) Paths
Taxonomic Extent

LTER Evaluation
Evaluation level metadata should include detailed descriptions of the project, methods, protocols, and intellectual rights in order for a potential user to evaluate the relevance of the data package for their research study or synthesis project.

Resource Use Constraints
Information about how the data may or may not be used after access is granted to assure the protection of privacy or intellectual property. This includes any special CSDGM /metadata/idinfo/useconst EML /eml:eml/*/intellectualRights restrictions, legal prerequisites, terms and conditions, and/or limitations on using the data set. Data providers may request acknowledgement of the data from users and claim no responsibility for quality and completeness of data.

Process
Step A step in the processing that produced a resource

LTER Access
Access-level metadata should provide a user with all the information needed to access and download the data tables, even if the tables' attributes are not thoroughly described. The tags required at this level specify access control and the physical description of the

LTER Integration
Integration-level metadata should support computer-mediated access and processing of data, and therefore requires that all aspects of the data package be fully described.