We study existence of solutions for the fractional Laplacian equation in , , with critical exponent , , , where has a potential well and is a lower order perturbation of the critical power . By employing the variational method, we prove the existence of nontrivial solutions for the equation. 1. Introduction In the last 20 years, the classical nonlinear Schr？dinger equation has been extensively studied by many authors [1–10] and the references therein. We just mention some earlier work about it. Brézis and Nirenberg [1] proved that the critical problem with small linear perturbations can provide positive solutions. In [3], Rabinowitz proved the existence of standing wave solutions of nonlinear Schr？dinger equations. Making a standing wave ansatz reduces the problem to that of studying a class of semilinear elliptic equations. Floer and Weinstein [10] proved that Schr？dinger equation with potential and cubic nonlinearity has standing wave solutions concentrated near each nondegenerate critical point of . However, a great attention has been focused on the study of problems involving the fractional Laplacian recently. This type of operator seems to have a prevalent role in physical situations such as combustion and dislocations in mechanical systems or in crystals. In addition, these operators arise in modelling diffusion and transport in a highly heterogeneous medium. This type of problems has been studied by many authors [11–18] and the references therein. Servadei and Valdinoci [11–14] studied the problem where , is an open bounded set of , , with Lipschitz boundary, is a real parameter, and is a fractional critical Sobolev exponent. is defined as follows: Here is a function such that there exists such that and for any . They proved that problem (1) admits a nontrivial solution for any . They also studied the case and , respectively. Felmer et al. [15] studied the following nonlinear Schr？dinger equation with fractional Laplacian: where , , and is superlinear and has subcritical growth with respect to . The fractional Laplacian can be characterized as , where is the Fourier transform. They gave the proof of existence of positive solutions and further analyzed regularity, decay, and symmetry properties of these solutions. In this paper, we consider the following problem: with critical exponent , , , where has a potential well, where is the fractional Laplace operator, which may be defined as is the usual fractional Sobolev space. is a lower order perturbation of the critical power . Now we give our main assumptions. In order to find weak solutions of (5),
H. Brézis and L. Nirenberg, “Positive solutions of nonlinear elliptic equations involving critical Sobolev exponents,” Communications on Pure and Applied Mathematics, vol. 36, no. 4, pp. 437–477, 1983.
M. Clapp and Y. Ding, “Positive solutions of a Schr？dinger equation with critical nonlinearity,” Zeitschrift fur Angewandte Mathematik und Physik, vol. 55, no. 4, pp. 592–605, 2004.
M. B. Yang and Y. H. Ding, “Existence of solutions for singularly perturbed Schr？dinger equations with nonlocal part,” Communications on Pure and Applied Analysis, vol. 12, no. 2, pp. 771–783, 2013.
M. B. Yang, “Existence of solutions for a quasilinear Schr？dinger equation with subcritical nonlinearities,” Nonlinear Analysis: Theory, Methods & Applications, vol. 75, no. 13, pp. 5362–5373, 2012.
H. Brézis and L. Nirenberg, “Positive solutions of nonlinear elliptic equations involving critical Sobolev exponents,” Communications on Pure and Applied Analysis, vol. 36, no. 4, pp. 437–477, 1983.
J. Chabrowski and J. Yang, “Existence theorems for the Schr？dinger equation involving a critical Sobolev exponent,” Zeitschrift fur Angewandte Mathematik und Physik, vol. 49, no. 2, pp. 276–293, 1998.
A. Floer and A. Weinstein, “Nonspreading wave packets for the cubic Schr？dinger equation with a bounded potential,” Journal of Functional Analysis, vol. 69, no. 3, pp. 397–408, 1986.
R. Servadei and A. Brézis-Nirenberg, “A Brezis-Nirenberg result for non-local critical equations in low dimension,” Communications on Pure and Applied Analysis, vol. 12, no. 6, pp. 2445–2464, 2013.
P. Felmer, A. Quaas, and J. G. Tan, “Positive solutions of nonlinear Schr？dinger equation with fractional Laplacian,” Proceedings of the Royal Society of Edinburgh, vol. 142, no. 6, pp. 1237–1262, 2012.
E. Di Nezza, G. Palatucci, and E. Valdinoci, “Hitchhiker's guide to the fractional Sobolev spaces,” Bulletin des Sciences Mathematiques, vol. 136, no. 5, pp. 521–573, 2012.
R. Servadei and E. Valdinoci, “Mountain Pass solutions for non-local elliptic operators,” Journal of Mathematical Analysis and Applications, vol. 389, no. 2, pp. 887–898, 2012.
S. Palatucci, G. Dipierro, and E. Valdinoci, “Existence and symmetry results for a Schr？dinger type problem involving the fractional Laplacian,” Le Matematiche, vol. 68, no. 1, pp. 201–216, 2013.