IPUMS

Documentation for IPUMS.

IPUMS.jl Demo

IPUMS.jl is an in-development package built on OpenAPI.jl for accessing IPUMS data via their API.

Downloading Data from NHGIS

Set up the API client using your key from the NHGIS IPUMS website, then point to an extract definition file.

using IPUMS

api = IPUMSAPI("https://api.ipums.org/", Dict("Authorization" => "Your_Key"))

Submit an extract request:

test_extract_definition = "test/testdata/example_extract_request.json"

Check the extract's status:

res = extract_submit(api, "nhgis", test_extract_definition)
metadata, defn, msg = extract_info(api, res.number, "nhgis")

List all previously submitted extracts:

extract_list(api, "nhgis")

Download the result to a local path.

extract_download(api, res.number, "nhgis"; output_path = "file_downloads/")

Loading CPS Extract Data

Parse the DDI metadata file and load the extract into a DataFrame using sample data from the IPUMS.jl repo.

ddi = parse_ddi("test/testdata/cps_00157.xml")
df = load_ipums_extract(ddi, "test/testdata/cps_00157.dat")

The resulting DataFrame looks like this:

3×8 DataFrame
 Row │ YEAR   SERIAL  MONTH  ASECWTH  STATEFIP  PERNUM  ASECWT   INCTOT
     │ Int64  Int64   Int64  Float64  Int64     Int64   Float64  Int64
─────┼─────────────────────────────────────────────────────────────────────
   1 │  1962      80      3  1475.59        55       1  1475.59       4883
   2 │  1962      80      3  1475.59        55       2  1470.72       5800
   3 │  1962      80      3  1475.59        55       3  1578.75  999999998

IPUMS.DDIInfo — Type

DDIInfo(
    filepath::String, 
    conditions::String = "", 
    citation::String = "", 
    ipums_project::String = "",
    extract_notes::Sring = "", 
    extract_date::String = "",
    variable_info::Vector{DDIVariable} = DDIVariable[]
    _xml_doc::EzXML.Document = EzXML.XMLDocument()
    _ns::String = ""
    data_summary::DataFrame = DataFrame()
)

A struct representing the metadata taken from an IPUMS extract. An IPUMS extract contains both file-level metadata (such as the date of export), as well as variable level metadata (such as the name and data type of a variable).

The DDIInfo object is not generally called directly. The parse_ddi() function creates a DDIinfo object after successfully parsing a DDI file from an IPUMS extract.

The DDIInfo object contains file level metadata. The variable_info field of the DDIInfo object contains a vector of DDIVariable objects. DDIVariable objects contain metadata information about individual IPUMS variables.

Keyword Arguments

filepath::String - File system path to the DDI (.xml) file.
conditions::String - IPUMS legal specification on the proper use of IPUMS data.
citation::String - Information for the citation of IPUMS data.
ipums_project::String - Identifier for the IPUMS source of the extract data, such as IPUMS CPS, or IPUMS USA, etc.
extract_notes::String - Additional clarifying information or user nodes about the extract.
extract_date::String - Date on which the extract was produced.
variable_info::Vector{DDIVariable} - a vector of DDIVariable objects, which contain metadata on each variable or column in the data file.
_xml_doc::EzXML.Document - An internal attribute that contains an internal representation of the DDI DOM for parsing.
_ns::String - An internal attribute to hold any namespaces used in the XML DOM.
data_summary::DataFrame - Contains a dataframe that holds summary information about the variables in the dataset, including variable names, data types, variable descriptions, and categorical information.

Returns

DDIInfo object that contains both file-level and variable-level metadata extracted from an IPUMS DDI (.xml) file.

Example

julia> IPUMS.DDIInfo(filepath = "test_ddi.xml")

IPUMS.DDIInfo("test_ddi.xml", "", "", "", "", "", IPUMS.DDIVariable[], EzXML.Document(EzXML.Node(<DOCUMENT_NODE@0x00000000034466d0>)), "", 0×0 DataFrame)

References

Information about each variable field is taken from:

https://ddialliance.org/Specification/DDI-Codebook/2.5/XMLSchema/fieldleveldocumentationfiles/schemas/codebookxsd/elements/var.html