In an era of widespread health misinformation and inequities, the ability to critically evaluate health information is increasingly crucial. Unfortunately, many individuals lack the skills to assess the reliability of and evidence behind health claims. There is an urgent need to understand and characterize the U.S. public’s ability to think critically about health and treatments. This study addresses this gap by being the first to psychometrically validate items from the Claim Evaluation Tools item bank – a tool for assessing health-related critical thinking in the public – with a U.S. sample. Using Rasch analysis, we examined 55 multiple-choice questions addressing Informed Health Choices Key Concepts, with a sample of 370 participants. Findings show promise and point to future directions for a validated tool for evaluating the U.S. public’s health-related critical thinking. This tool can be used to evaluate interventions in the U.S. aimed at enhancing critical health literacy and empowering the public to make informed healthcare decisions.