This might not really be the sort of maths you’re looking for, but another way of getting a look at mathematical vocabulary in Japanese would be to look for sites that discuss entrance examination preparation. I came across a question in an anime that I couldn’t solve, only to find that it was a 2012 University of Tokyo entrance exam question. I then found a solution for it on a website that explained how to solve it step by step. I don’t really feel like looking for it right now, but I think the maths problem appeared in Season 2 of the Nisekoi anime in a study session scene, and I believe that just searching the problem text brought me to the website. Such pages are a good way to learn how mathematical reasoning is presented in Japanese, using such words as
AをBと置く – let A be B
n乗 – nth power
自然数 – natural number
整数 – whole number
EDIT FOR CLARITY: by that, I mean the etymological sense of ‘integer’, which means ‘intact’ or ‘whole’ in Latin, and as such includes all numbers without a fractional or decimal part, including negative ones. This is the set written as \mathbb{Z} in mathematical notation. However, it seems 整数 is ambiguous in both Japanese and Chinese. 整数 can mean both ‘positive whole number greater than or equal to zero’ and ‘integer’.
示す – show/prove
and so on.
EDIT: OK, turns out it was easy to find that site. Here you go: