I think it’s best to think of toshokan as a existing word in spoken conversation. Kanji – letters, are just used to guess how to read signs.
Actually it can be broken down, just not in the way you think
It doesn’t break arbitrarily into syllables, but into root words 図書 + 館. A Japanese children might be able to read としょ館, but that would be rare in adult books.
I am sure you can get library out of 図書 + 館.
It’s somewhat similar to a libra-ry, if you separate the final first.
How you think li-bra-ry is pronounced? How would pronounce “li” without context?
Maybe you can have some doubts with starting from Kanji and written characters.
A spoken quiz might be,
What building do you want to go to find a classic reference book?
How do you call “library” in Japanese?
What is that building you have walked past by?
They’re really just squiggles
Simple stuff are certainly highly contrived. Rarer stuff are less contrived. (and you can end up as bad at reading exceptional words as natives, don’t worry)
That being said, written materials are just hard in their own way, even if you might know some of the spoken Japanese.
radicals are made up and only good for dictionary reference?
Another thing to keep in mind, is that Kanji may get simplified or altered during the thousands of years, and it may lose connection to their parts. 館 is generally just a building, not having anything to eat.
In Chinese, it’s simplified differently, 圖書館 / 图书馆. So indeed, radicals are made up here. People don’t agree with what is in the 口.