Measures of information processing speed vary between individuals and decline with age. Studies of aging twins suggest heritability may be as high as 67%. The Illumina HumanExome Bead Chip genotyping array was used to examine the association of rare coding variants with performance on the Digit-Symbol Substitution Test (DSST) in community-dwelling adults participating in the Cohorts for Heart and Aging Research in Genomic Epidemiology (CHARGE) Consortium. DSST scores were available for 30,576 individuals of European ancestry from nine cohorts and for 5758 individuals of African ancestry from four cohorts who were older than 45 years and free of dementia and clinical stroke. Linear regression models adjusted for age and gender were used for analysis of single genetic variants, and the T5, T1, and T01 burden tests that aggregate the number of rare alleles by gene were also applied. Secondary analyses included further adjustment for education. Meta-analyses to combine cohort-specific results were carried out separately for each ancestry group. Variants in RNF19A reached the threshold for statistical significance (p = 2.01 × 10−6) using the T01 test in individuals of European descent. RNF19A belongs to the class of E3 ubiquitin ligases that confer substrate specificity when proteins are ubiquitinated and targeted for degradation through the 26S proteasome. Variants in SLC22A7 and OR51A7 were suggestively associated with DSST scores after adjustment for education for African-American participants and in the European cohorts, respectively. Further functional characterization of its substrates will be required to confirm the role of RNF19A in cognitive function.