Just as Toxite mentioned, if you’re new to computer vision, jumping in to make modifications to the model structure might not be the best idea. It might be a better use of time to learn more about the fundamentals of computer vision and convolutional neural networks structure first.
I like to use an analogy to help understand why I, and others make this type of recommendation. As a child, once you start walking, you don’t immediately try to run a marathon. In fact, even after becoming a teenager or adult, you don’t just run a marathon, you have to work up to it with lots of training. It’s tempting to skip the fundamentals, but there’s a reason they’re called ‘fundamentals’ and that’s why they shouldn’t be skipped.
Finally, I also recommend you give this post a read. You’ve shared what you’re trying to do, but you haven’t provided the context as to the why. Without explaining what you aim to accomplish, it’s impossible to give you assistance with what your true goal is. Generally, most people have a reason to modify a model, but we don’t know why you want to, and it might be that for what you actually want to accomplish, you don’t need to. You could save yourself a lot of time by sharing more information as to what you are attempting to achieve overall, instead of just what the problem is you’re trying to tackle right now.