Overview
The Assessment Data Scientist utilizes knowledge of learning sciences, assessment design, analytics, measurement and educational data mining to realize the potential of games and digital learning experiences for engagement, learning and assessment. The person in this role collaborates with clients, internal teams and external partners to apply and create robust approaches to assessment design and data analysis while ensuring the accuracy, utility and accessibility of interpretations. Lastly, the Learning Data Scientist provides support, when needed, to identify and develop external partnerships with leading figures in the fields of learning analytics, educational data mining and intelligent systems for purposes of joint projects that advance the work in the field of digital, game-based learning and assessment tools.
Primary Duties and Responsibilities
Assessment Design
● Contribute to the design of internally developed and partner created content by articulating learning outcomes based on reviews of relevant literature and informational calls with leading researchers and practitioners,
● Contribute to iterative learning content testing by designing and implementing playtest and mini-tryout sessions with members of target agegroups, as well as designing and carrying out interview and think aloud protocols;
● Apply an Evidence Centered Design (ECD) or related framework to articulate task, evidence and student models within interactive digital learning experiences.
Data Flow and Analysis
● Collaborate with external partners and internal teams to improve and maintain a complete data pipeline from initial data generation within digital learning experiences through multiple stages of cleaning, parsing, analysis, reporting and archiving;
● Employ a range of tools to clean, parse and annotate telemetry;
● Apply data mining and learning analytic techniques, and more traditional statistical models where needed, to investigate, identify and confirm relevant patterns of behavior within games and digital learning experiences in order to support inferences about learners’ cognitive and affective states;
● Support success of internal content and third party learning content, integrating knowledge of analytics, learning and assessment design to make datadriven recommendations for improving digital learning and assessment tools.
● Lead the seamless integration of learning analytics and evidence centered design informed learning design to support iteration and improvement of games and digital learning experiences for learning and assessment and play a key role in the success of the organization’s services to external game developers.
Research and Partnership Development
● Contribute to advancement of the organization’s research agenda.
● Initiate and cultivate communications and partnerships with leading experts in the areas of learning analytics, educational data mining and intelligent systems with the aim of growing the organization’s capacity for developing and applying high leverage approaches to making generalizable inferences about learner abilities, cognition, affect and other states.
● Investigate and evaluate novel approaches to working with telemetry from game-based and interactive digital assessments and learning tools in order to incorporate the most promising approaches and grow the organization’s skill base;
● Act as a ‘voice’ for the organization’s analytics, presenting the team’s approaches to data mining and analytics to external audiences;
● Contribute to fundraising efforts as needed.
Minimum Qualifications
– Expertise in data analytics, including data mining, learning analytics, and statistical modeling;
– Competency in data analysis tools, including the R programming language (RStudio), and data mining software (e.g. NumPy, Pandas, scikitlearn, RapidMiner, WEKA);
– Experience in handling telemetry data, including developing expedient processes for cleaning and aggregating data on the student level;
– Facility in designing data structures to capture a consistent, scalable schema of data across digital learning contexts;
– Ability to select base data features relevant to a given analysis, aligned with the data capture schema above; building on this, the ability to engineer secondary features (e.g. standard deviation of time elapsed) based on base data features (e.g. total time elapsed);
– Ability to effectively communicate complex concepts to a lay audience.
– Ability to collaborate effectively with a diverse team.
– Comfortable working in fast-paced dynamic environment focused on product
development.
Preferred Qualifications
The ideal candidate will also have the following preferred prior experience, skills and dispositions:
● Knowledge of Evidence Centered Design, basic psychometrics, and research design
● Ability to apply Evidence Centered Design (ECD) or similar framework to design, create, test and revise assessments within interactive digital learning content,
● Experience in learning progressions, learning sciences, and/or game-based learning
● Familiarity with Connected Learning and/or research in Digital Media and Learning (DML)
● Prior knowledge and/or willingness to learn programming basics as part of work with tech team and data analysis
Education
– Required: Master’s Degree or equivalent in data sciences, learning sciences, or related field
– PhD preferred
About GlassLab, Collective Shift, and LRNG
GlassLab’s background is designing and developing high impact digital games, interactive learning tools, and deep analytics, which make learning visible for underserved youth and their teachers, coaches and parents. GlassLab is currently expanding its work by merging with a new company called Collective Shift formed by the MacArthur Foundation. Collective Shift’s initial focus will be on LRNG, based on Connected Learning, a new pedagogy that emerged from nearly $150 million in research and demonstration projects funded by the MacArthur Foundation over the past decade. Connected Learning calls for learning that is based on kids’ interests, socially networked and relevant to the real world. A key component of the endeavor is Cities of LRNG, which will build on the success of Cities of Learning, a three year demonstration project. Cities of Learning in Chicago, Dallas, Pittsburgh and Washington, D.C., served more than 100,000 youth ages 4 to 24 during recent summer programming.
Together, GlassLab and Collective Shift will bring the LRNG Vision and Mission to life.
GlassLab, Inc. is an equal opportunity employer and does not discriminate on the basis of age, disability, sex, gender identification, sexual orientation, genetic information, national origin, race, religion or any other protected class.
Note to Third Party Agencies
GlassLab does not accept unsolicited resumes from Third Party Agencies. GlassLab will not consider or approve payment regarding recruiter fees or referral compensations. In the event a recruiter or agency submits a resume or candidate without a previously signed agreement, GlassLab explicitly reserves the right to pursue and hire those candidate(s) without any financia responsibility to the recruiter or agency.
How to apply
Please send an email with the subject “Assessment Data Scientist” and attach a cover letter and resume to: jobs@collectiveshift.org.