Testing GPT-4.0 accuracy rate on Japanese language

Japanese Language
161

It mostly stitched together sentences from the original article no? There is not a lot of room to get wrong.

If you want some example in which GPT4-0 goes off the road, here are two of them. It’s pretty representative of what happen in more difficult sentences IMHO, ChatGPT will say many right things, many wrong things, all with an authoritative tone.

From the senryuu thread:

Translate the following senryuu, break down the grammar and explain the humor: お噂は聞いていますと吹き出され

ChatGPT

This Japanese senryū translates to: “I’ve heard the rumors,” you said, making me burst into laughter.

Now let’s break down the grammar:

  1. お噂 (o uwasa): This translates to “rumor”. In Japanese, “o” is an honorific prefix that makes the following word more formal or polite.
  2. は (wa): This is a particle used after the topic of the sentence, in this case, “rumor”.
  3. 聞いています (kiiteimasu): This is the present continuous form of the verb “to hear” or “to listen” (聞く - kiku). This implies that the action (listening/hearing) is currently ongoing.
  4. と (to): This is a quoting particle that is often used before verbs like “say”. In this context, it’s used to indicate what the person said.
  5. 吹き出され (fukidasare): This is the passive form of the verb “to burst into laughter” or “to spurt out” (吹き出す - fukidasu). The use of passive voice here emphasizes the action’s impact on the speaker.

Now, regarding the humor: The humor in this senryū relies on the implicit contradiction or irony. The act of someone mentioning they’ve been hearing rumors isn’t usually something to make one laugh out loud. However, the laughter could suggest the speaker finds the situation absurd, maybe because the rumors are wildly inaccurate, or because they find it amusing that the other person believes them, or perhaps even because the rumors are actually true but ridiculous. It’s up to the reader’s interpretation.

It’s rare, but even the English translation seems a bit weird to me. "I've heard the rumors”, you said, making me burst into laughter. Who is laughing here?

It’s even more confusing with the explanation about the passive: The use of passive voice here emphasizes the action's impact on the speaker. Who is “the speaker” ? The person telling the senryuu ? The person saying “お噂は聞いています” ? Are they the same? (In reality it’s a suffering passive, someone said “I’ve heard a lot about you” to the senryuu author while doing a 吹き出す)

This implies that the action (listening/hearing) is currently ongoing. The use of “currently ongoing” is a bit unclear no? It make it sounds like it’s a ている about what is going on right now, like how 食べている can mean “I’m currently eating”.

The entire explanations about the humor is completely wrong and again it’s very unclear who ChtGPT think is doing the 吹き出す.

Please translate the following sentence and break down the grammar: そりゃ話に乗ってくると取っていいんだな？

ChatGPT

The Japanese sentence “そりゃ話に乗ってくると取っていいんだな？” can be translated into English as “If they go along with the story, it’s okay to take it, right?”.

Let’s break down the grammar:

  • そりゃ: This is a colloquial contraction of “それは”, which means “that is”. However, in this context, it’s being used as a filler word, similar to how “well” or “you see” might be used in English.
  • 話に乗ってくる: The term “話” means “story” or “talk”, “乗る” means “to ride” or “to board”, but in this context, it is used metaphorically to mean “go along with” or “agree to”. The “に” particle indicates the direction or target of the action, and “くる” is a helper verb that implies the action is coming towards the speaker. So “話に乗ってくる” means “to go along with the story”.
  • と: This is a conditional particle that is similar to “if” or “when” in English.
  • 取って: This is the te-form of the verb “取る”, which means “to take”. The te-form of a verb is often used to connect two clauses or to express a conditional relationship.
  • いい: This adjective means “good” or “okay”. When combined with the te-form of a verb, it’s used to express permission or acceptance, similar to the English phrase “it’s okay to”.
  • んだ: This is a contraction of “のだ”, which is a sentence-ending phrase used to provide explanation or emphasis. Here, it can also imply that the speaker is seeking confirmation or agreement from the listener.
  • な?: This is a sentence-ending particle used to seek confirmation or agreement from the listener, similar to “right?” or “isn’t it?” in English.

So, putting it all together, “そりゃ話に乗ってくると取っていいんだな？” translates to “If they go along with the story, it’s okay to take it, right?”.

Not completely wrong, it correctly say that 話に乗る is an idiom, but the sentence is supposed to mean something like "So can I take that to mean you agree / you are in ? (about something they talked earlier, probably)

と: This is a conditional particle that is similar to "if" or "when" in English. Doubtful, seems more like the quotation と to me.

5 Likes
162

I lack the technical knowledge to interpret your explanation as well, I’m afraid :joy: in context of deep learning, what are weights? Neurons? What’s an initial activation? Bias? (I know what it is in psychology of cognitive processes)
Appreciate your effort anyway :slightly_smiling_face:

I’ve been seriously considering of beginning to study computer science (as soon as I’m able to build the prerequisites…). Can I ask you a suggestion?

163

No, I appreciate it!

164

We know GPT4 can go quite wrong with very little to do haha

165

I think if you had it visually laid out before you it would be a lot easier honestly. The video I linked does a very good job.

So think like in your brain how you have brain cells connected to a bunch of other brain cells. When one cell lights up, it sends a signal to the other cells it’s connected to, right? That cell that lit up is the initial activation. But it doesn’t just send the same signal everywhere, or else that wouldn’t be very useful. Some of the cells it’s connected to are with a really strong connection and some have a really weak connection. So if it lights up a little bit, but has a strong connection with a cell, it will send a normal sized signal still. But if it lit up a little bit, and had a very weak connection, it would send a super weak signal.

Like let’s use some nice numbers. Assume the strength of connection is a nice .5 and The initial cell lit up with a brightness of 8.

. 5 * 8 =4. It sends a signal of 4 to the next cell and it will light up with a brightness of 4 now.

If the initial cell lit up with a brightness of .2 instead but we had a strength of 10, then:

10 *.2 =2 so it sends a signal of 2 to the next guy in line.

The initial brightness of the cell that fired is the activation value and the strength of connection (the .5 and 10) is the weight. In practice, they have multiple connections to each neuron, so you would add all the incoming signals together and many neurons would be doing this in parallel in what’s called a “layer”. And what you just saw where I calculated out the signals that would be sent to the next cell and determine the activation of the next cell in line is called forward propagation.

4 Likes
166

I gave it a brief check and it was very simple and well presented especially graphically but I’m having some eyesight issues today probably because of pollen + screen time increase and couldn’t see shit for the whole afternoon…

I got your explanation, very intuitive, but I’m missing what is the bias in your previous explanation.

But now I partially understand this:

Out of curiosity, you have a strong mathematical background? How much do you know of CS? I’d like to make you a couple of specific questions. I posted some on reddit, and luck would have it that today morning many subreddits (including math and computer science) obscured as a protest to the new upcoming regulations on API if I’m not wrong

167

IMG_4048
IMG_40481240×914 202 KB

(Original source) Someone was wondering how wolfram is supposed to improve gpt-4 at languages if remember good… perhaps this makes sense for someone? :joy:

168

So a bias is just some value added on at the very end.

Take an activation of brightness 10 and a weight of 3.

10*3=30 so the signal strength is 30. If there was a bias of… Say 20, we would add 20 to this 30 and be left with a final signal of 50 that lights up the next neuron.

What this is used for depends on if it’s positive or negative. You can think of adding a big negative number as making it to where the signal needs to be really strong or else it will just get canceled out. If it’s positive, it’s like helping that second neuron light up.

One of my degrees is in math so I’m pretty comfortable with it. Lot of stuff I don’t know, though. I did CS for a few years in hs and college and that was actually my initial major before I changed it, and I’ve been doing it more recently as well but a lot of that has been just teaching myself web development. My current project just happens to be something that uses a CNN that I had to develop myself, so that coupled with general interest has had me learning about this stuff more than normal.

You’re free to ask away, but depending on what it is I may do more redirection towards better sources than answering myself since I try to not fall into the trap of overconfident beginners.

4 Likes
169

Perfectly clear, only incognita now is what determine biases and their measure. Is it weight related?

Wow, what is the other?

Learn, you must, hmm? (In case you didn’t get it, this is Yoda speech).
You have a peculiar syntax at times, what’s your native language if I can ask?

Mine is a simple question actually.

simple question

As I mentioned, I just begun thinking seriously about getting into CS. I don’t know if I’ll just study it as a self learner or I’ll apply to college one day, and at the moment I don’t even care.
The point is, I’m totally ignorant at mathematics. I don’t mean that I’m not good at it, I believe I’m quite good at understanding and visualizing concepts at an intuitive level (as a math degree you can laugh at me if you want). But it’s been so long since I last applied any mathematical concept that I forgot everything. I will probably have to study it starting from middle school level…
I’m not concerned by difficulty level and time required, one year ago (right before beginning to study japanese) I started with powers and roots all the way through second grade inequalities and basic functions in 2.5 months and it was easy-ish with occasional obstacles. The house of cards eventually fell because I had completely neglected geometry, trigonometry and logic, and more complex functions started not to make sense anymore.
This was fault of the complete lack of a study roadmap, I was basically studying everything in the order a website suggested it, but it wasn’t tailored for any goal whatsoever.
Now that I have access to GPT-4 I came up with the idea of asking it to build a roadmap that can bring me from zero knowledge up to some solid foundations that allow me to begin studying CS. I worked with the prompt and asked it to make the study plan’s structure propedaeutical and don’t exclude important concepts nor fill it with too many useless ones.

This is what it got me:

roadmap

1.Basic Mathematics

1.1. Arithmetic

1.1.1. Addition: Properties of addition, adding real numbers

1.1.2. Subtraction: Properties of subtraction, subtracting real numbers

1.1.3. Multiplication: Properties of multiplication, multiplying real numbers

1.1.4. Division: Properties of division, dividing real numbers

1.1.5. Fractions: Simplifying fractions, operations with fractions

1.1.6. Decimals: Converting between fractions and decimals, operations with decimals

1.1.7. Powers and Roots: Properties of powers, simplifying expressions with powers, square roots, cube roots

1.2. Algebra

1.2.1. Variables and Expressions: Evaluating expressions, simplifying expressions

1.2.2. Equations: Solving linear equations, solving quadratic equations, solving rational equations

1.2.3. Inequalities: Solving inequalities, compound inequalities, absolute value inequalities

1.2.4. Functions: Function notation, domain and range, types of functions (linear, quadratic, exponential)

1.3. Geometry

1.3.1. Shapes: Properties of shapes, perimeter and area of shapes

1.3.2. Angles: Types of angles, angle relationships, measuring angles

1.3.3. Theorems: Pythagorean theorem, circle theorems

1.3.4. Volume: Volume of prisms, cylinders, pyramids, cones, spheres

1.4. Trigonometry

1.4.1. Sine, Cosine, Tangent: Definitions, SOHCAHTOA, reciprocal trigonometric functions

1.4.2. Trigonometric Identities: Pythagorean identities, double-angle identities, half-angle identities

1.4.3. Unit Circle: Definition, trigonometric functions on the unit circle

1.4.4. Trigonometric Equations and Inequalities: Solving trigonometric equations, solving trigonometric inequalities

1.5. Formal Logic

1.5.1. Propositional Logic: Logical connectives, truth tables, tautologies and contradictions, logical equivalence

1.5.2. Predicate Logic: Quantifiers, predicates, logical equivalence in predicate logic

2.Advanced Mathematics

2.1. Calculus

2.1.1. Limits: Definition of limit, properties of limits, finding limits, continuity

2.1.2. Derivatives: Definition of derivative, rules of differentiation, applications of derivatives

2.1.3. Integrals: Definition of integral, rules of integration, applications of integrals

2.1.4. Series: Convergence tests, power series, Taylor and Maclaurin series

2.2. Complexity and Algorithmic

2.2.1. Time Complexity: Big O notation, time complexity of common algorithms

2.2.2. Space Complexity: Memory usage of algorithms, trade-off between time and space complexity

2.2.3. Algorithmic Paradigms: Greedy algorithms, divide and conquer, dynamic programming

2.3. Multivariable Calculus

2.3.1. Multiple Integrals: Double and triple Integrals, applications

2.3.2. Vector Calculus: Vector fields, line integrals, applications

2.3.3. Theorems: Green’s theorem, Stokes’ theorem, Divergence theorem

2.4. Linear Algebra

2.4.1. Matrices: Operations with matrices, determinant, inverse matrix

2.4.2. Vectors: Operations with vectors, dot product, cross product

2.4.3. Spaces: Vector spaces, subspaces, basis and dimension

2.4.4. Transformations: Linear transformations, eigenvalues and eigenvectors

2.5. Differential Equations

2.5.1. First Order: Separable, linear, exact differential equations

2.5.2. Second Order: Homogeneous, non-homogeneous, systems of differential equations

2.5.3. Partial: Heat equation, wave equation, Laplace’s equation

2.5.4. Ordinary: Initial value problems, boundary value problems

2.6. Numerical Analysis and Optimization

2.6.1. Numerical Optimization: Gradient descent, Newton’s method, linear programming

2.6.2. Error Analysis: Round-off errors, truncation errors

2.6.3. Numerical Linear Algebra: Matrix factorizations, direct and iterative methods for linear systems

2.6.4. Numerical Differential Equations: Euler’s method, Runge-Kutta methods

3.Probability and Statistics

3.1. Descriptive Statistics

3.1.1. Measures of Central Tendency: Mean, median, mode

3.1.2. Measures of Dispersion: Variance, standard deviation, interquartile range

3.1.3. Graphical Representations: Histograms, box plots, scatter plots

3.2. Probability Theory

3.2.1. Basic Probability: Sample spaces, events, probability axioms

3.2.2. Conditional Probability: Definition, Bayes’ theorem

3.2.3. Random Variables: Discrete and continuous random variables, probability mass function, probability density function

3.2.4. Probability Distributions: Uniform, binomial, normal distributions

3.3. Inferential Statistics

3.3.1. Hypothesis Testing: Null and alternative hypotheses, p-value, type I and II errors

3.3.2. Confidence Intervals: Interpretation, construction for means and proportions

3.3.3. ANOVA: One-way and two-way ANOVA, assumptions, interpretation

3.4. Regression Analysis

3.4.1. Simple Linear Regression: Least squares method, coefficient of determination, hypothesis tests for regression

3.4.2. Multiple Linear Regression: Interpretation of coefficients, multicollinearity, interaction effects

3.4.3. Logistic Regression: Odds ratio, maximum likelihood estimation, model fit

Does it make sense? Would you change anything?

170

There are math classes in CS, for the important stuff. And I’d say depending on what you want, you might not need that much math for just CS, more like logic and problem-solving. But if you’re interested in this, then math is probably a good idea.

I’m not a math expert or anything, and I don’t remember having trouble with it in university. But of course, it depends on courses and what you learn.

CS can easily be learned online, and most people do (and sometimes have to) learn a bunch online even if going to a school.

Like you don’t need geometry or trigonometry, not that it’s hard, but you can easily get lost in trying to become an expert in basic math instead of just knowing the surface level and figuring it out as you go, which is very very common in CS.

I guess it’d be easier to ask what you think CS is and what you want to do with it. Or maybe what program you’d be doing.

2 Likes
171

Bias is separately determined. The forward pass is basically what I summed up above, but what actually determines their values is something called backpropogation that takes place during training. Basically, we initialize the weights and biases to be random, and then give the model a training problem. It’s very likely to do terrible since it’s all random weights, but that’s OK. Backpropogation is basically using a bit of calculus to see what it got wrong and walk back through the neurons and nudge their weights and biases in the right direction so they would have done a better job at giving us the output we want. We repeat this process a lot and through a lot of small nudges we arrive at our final weights and biases that we want to keep.

Secondary education. I have my teaching license and can teach high-school math.

Its English haha. Perhaps I naturally have a peculiar way of talking, but I think my ability has sort of become muddied the past year or so. Most weeks nowadays I don’t actually speak any English in normal conversations. So maybe that + Japanese thinking is creating weird syntax at times.

I think I might be able to give a bit of input but id rather take the time to think and carefully type out an answer rather than quickly here during a break at work like I’m doing right now. I’ll try to reply to this during my lunch break!

1 Like
172

Well, the roadmap is definitely a nice one that seems normal for maybe people in college. Some places are weird in my opinion in some ordering, especially around the end. Stokes theorem and triple integrals is much less important imo than basic linear algebra, for example, but its not uncommon for that ordering in college since iirc triple integrals are calc 3 and plenty of people do that before matrix algebra/linear algebra.

My personal take, however, is that the premise of question is kinda wrong from the get go. I don’t think you need to really learn any math outside of addition, subtraction, multiplication, and division to get started in cs. I can’t speak about further down the road, but from zero to getting where I’m at, its a lot more about problem solving skills and thinking like a programmer than any background knowledge in math. Then, when you run into problems that require math, you can learn more about it. But just doing it as a hobby I think you’re ready to get started honestly.

I feel like people who tend to be good at math tend to be good at cs because being good at math on an innate level requires the ability to see patterns in how you can use more fundamental operations in combination with eachother to represent or solve problems. CS is similar in some ways and I think that similarity is why math people are good at CS and not because CS requires a lot of complicated math. Im not sure what area you want to get into, but if I had to describe the regular programming I have done it would just be summed up as “writing a really long and specific list of instructions that when followed exactly solve a problem”. The hard part in my opinion is more often coming up with the instructions I want to write rather than the actual process of writing them out in code. And those instructions usually don’t include complicated math, but rather simple math used in sometimes creative ways. Not to say that there isn’t harder math involved in CS, because there is, but your question was about knowledge to get started in CS and for that I really think the barrier to entry is just your 4 basic operations and “greater than” and “less than” maybe.

To leave you with an example of what I mean about coming up with instructions to solve a problem:

Sorting is a very common task in CS. There are a lot of very nice algorithms out there that can sort a list of numbers, but try thinking of how you would do it yourself. E.g. [1,4,5,2,3] ->[1,2,3,4,5]

So youre sitting across a table from someone looking at a list of 5 numbers. They understand “greater than” and “less than” and can swap the position of the numbers. What set of instructions could you give so that they could sort any list of 5 numbers?

EDIT: Since I realize I kinda conflated you saying you wanted to get into cs with you saying you want to program, just checking, what do you want to do? CS and programming are different though related as PM mentioned, and I assumed pretty boldly you meant the latter.

4 Likes
173

This is definitely true. But also, you don’t need that much CS to program a computer or be a software engineer. CS is an academic discipline at heart and isn’t always that closely connected to the job of getting a computer to do what you want in a reliable way. I use CS stuff maybe once a year in my day job, if that. (Though the background is helpful for spotting those once a year events so you don’t spend ages working on a problem that has a well known solution…)

3 Likes
174

This is very true, I kinda conflated CS with programming in my reply.

If you do want to go to college for a CS degree, you will need a lot more math than I outlined. My university required until Calc 2 + discrete math + linear algebra for CS students for reference.

3 Likes
175

Haha I was about to say something, but same :sweat_smile:.

@mariodesu as someone who’s a biologist by training, I can say taking CS is definitely a good idea. I have some pretty big gaps in math and basic computer science concepts and CS studies give a good foundation in both.

1 Like
176

Hey, thanks for the advice you and everyone else, it’s extremely appreciated and useful as well :pray:
Let me clarify that what I’m pursuing is LLMs and AI in general, and I’m not precisely going after programming. I imagine this will require quite more mathematics than purely programming…?

That’s the plan, I’ll use a mix of websites, video tutorials, communities and possibly GPT as well haha (very cautiously ofc but I heard that with the wolfram plugin it becomes very accurate at mathematics). I’m quite curious of the help it can provide at visualizing and ironing complex concepts.

About the last part, I’m just afraid that in this way I won’t understand much because I lack the basics… also, I read somewhere somewhere that geometry is not important for CS, but I also thought that some subjects made heavy use of trigonometry, end after a google search both geometry and trigonometry are required to understand calculus…?

What I think computer science is: study and planning of algorithms and data structures, software development.
What I want to do with it: provide my heroic contribute to ensure a safer terminator-proof world work around concepts I’m passionate about, mostly intelligence applied to computers.

Ok so bias(es?) and weights are factors that start randomized and gets assessed with trial and error (training). And also from what is my understanding now, while weights is a variable intrinsic of the neural connection, bias is it’s adjustment provided during training (so basically weight plus or menus bias)?

Turns out you were the best to ask to!!

177

Yep, that sounds like a good summary! And based off that assessment, they get little adjustments to make them better at what they are supposed to do.

While biases are optional, they aren’t just an adjustment provided during training. During actual use they are still active. You can think of a weight like the strength between neurons and bias like a little bonus that gets added (that can be negative).

To illustrate this with a simple example. Imagine we have a 2 neuron neural network. There is only 2 neurons so that gives us one connection. That connection will have a weight and bias. Lets say we work a job that pays 10 dollars per hour and our boss is nice and gives us 20 dollars just for showing up to work. And lets say we want to calculate how much pay we get for a day with our neural network. So if we input an initial activation of 4 (hours) into the first neuron, it should cause the second neuron to activate with our amount of money for the day. So if we put 4 (hours) into the first neuron we should get 4(hours) * 10 (dollars per hour) = $40 + bonus $20 for showing up = $60 dollars for the day! If we work 1 hour, we would get 1 (hour) * 10(dollars per hour) = $10 + bonus $20 for showing up = $30 dollars for the day!

So lets get our neural network ready and initialize random weight and bias. Ok so it looks like the weight is a 5 and the bias is a 1.

mspaint_oqr3Y5VrsW
mspaint_oqr3Y5VrsW596×152 1.25 KB

Cool! Lets train it by plugging in 4 hours. we know that this should give us $60 dollars out!

mspaint_xeLbE93ira
mspaint_xeLbE93ira567×213 1.86 KB

Wait, what? We only got a 21 out. Thats wayyy too low. Thats like 1/3 of what we were expecting. We need to nudge these weights and biases to make the second neuron light up more. Lets bump them up to 8 and 10 respectively. And lets give it 1 hour of work for training. We know that should result in $30.

mspaint_2PRbXqXLQk
mspaint_2PRbXqXLQk639×180 2.38 KB

Damn, $18 dollars. We got better, but its still not right. We got to keep nudging. This repetitive process will eventually get us to a weight of 10 and bias of 20 that will be the perfect solution.

1 Like
178

What’s your starting point by the way? As in, what’s your current major/minor?

Using any LLM for math sounds like a BadIdea™ by the way. It’s not about it being less or more accurate. It was not meant to do math so you’re always going to hit issues with computation.

1 Like
179

My concern is that it’s not just a couple of concepts to be laking, it’s the entire middle and high school math… and as I mention, my aim is AI and LLM. I suppose I’ll meet you on the point that those subjects require deeper math…?

Would you still suggest me to start approaching the most basic CS and studying math on the road, even knowing that I have such complex goals?

no action; less x2; less x2; more x2; more x2 ?

Would you still tell me the same after knowing that I’m not exactly aiming at the programming side of CS?

This is something I haven’t decided yet… my idea was: studying math by muyself, get to a “decent level” whatever it is and then choose wether to 1) apply to a college course; 2) even better, applying to a private course; 3) self studying and what happens happens but I suppose this way is harder to find job in the industry…? I imagine this world as a place where I can build stuff and use it to advertise myself (what was it, github?). But I guess some jobs (best jobs?) will always require some sort of certification, young age and high grades. Now that we’re talking about jobs, I’ll point out that I’m mainly doing this to pursue a an old inspiration and given my “old” age (27) I won’t have as much opportunities as a college student that just completed a master

Happy to hear this! What do you mean with biologist by training?

Sorry if the answerer is a bit confused but I’m suffering insomnia and haven’t slept in 48 hours :smiling_face_with_tear:

1 Like
180

It means that I don’t have an official education in computer science and/or coding. Had to learn everything myself while doing a PhD in biochemistry.

3 Likes