We study existence of solutions for the fractional Laplacian equation in , , with critical exponent , , , where has a potential well and is a lower order perturbation of the critical power . By employing the variational method, we prove the existence of nontrivial solutions for the equation. 1. Introduction In the last 20 years, the classical nonlinear Schr？dinger equation has been extensively studied by many authors [1–10] and the references therein. We just mention some earlier work about it. Brézis and Nirenberg  proved that the critical problem with small linear perturbations can provide positive solutions. In , Rabinowitz proved the existence of standing wave solutions of nonlinear Schr？dinger equations. Making a standing wave ansatz reduces the problem to that of studying a class of semilinear elliptic equations. Floer and Weinstein  proved that Schr？dinger equation with potential and cubic nonlinearity has standing wave solutions concentrated near each nondegenerate critical point of . However, a great attention has been focused on the study of problems involving the fractional Laplacian recently. This type of operator seems to have a prevalent role in physical situations such as combustion and dislocations in mechanical systems or in crystals. In addition, these operators arise in modelling diffusion and transport in a highly heterogeneous medium. This type of problems has been studied by many authors [11–18] and the references therein. Servadei and Valdinoci [11–14] studied the problem where , is an open bounded set of , , with Lipschitz boundary, is a real parameter, and is a fractional critical Sobolev exponent. is defined as follows: Here is a function such that there exists such that and for any . They proved that problem (1) admits a nontrivial solution for any . They also studied the case and , respectively. Felmer et al.  studied the following nonlinear Schr？dinger equation with fractional Laplacian: where , , and is superlinear and has subcritical growth with respect to . The fractional Laplacian can be characterized as , where is the Fourier transform. They gave the proof of existence of positive solutions and further analyzed regularity, decay, and symmetry properties of these solutions. In this paper, we consider the following problem: with critical exponent , , , where has a potential well, where is the fractional Laplace operator, which may be defined as is the usual fractional Sobolev space. is a lower order perturbation of the critical power . Now we give our main assumptions. In order to find weak solutions of (5),
H. Brézis and L. Nirenberg, “Positive solutions of nonlinear elliptic equations involving critical Sobolev exponents,” Communications on Pure and Applied Mathematics, vol. 36, no. 4, pp. 437–477, 1983.
P. Felmer, A. Quaas, and J. G. Tan, “Positive solutions of nonlinear Schr？dinger equation with fractional Laplacian,” Proceedings of the Royal Society of Edinburgh, vol. 142, no. 6, pp. 1237–1262, 2012.